From patchwork Fri Nov 24 10:18:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467436 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C2898C61D97 for ; Fri, 24 Nov 2023 10:20:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ieoKTYDHBZwXn6Y+u3AbGy4oTm+MxmhW/p3HSfBO6pQ=; b=S8TTd/UUQvxMQQVGwoLgoF4OXg L1xTFQYBInYtWvzV7BBqZJ+mET5/yBlVMPVYEsPXGrtsc6qbTuxYIpTjJ49KnhhlVe/OlIevOnQ1N BJ5j5wbvwkgD1gIDdFoD0wfbrLZz4dNJiP0OKLRWD8XGaxdk+kSkLFQsBhfzRGGnqX+er+2uegl+U oeRdyX7UcaegU1uLrNxpFruSOlxQnTAwl/7x0H7UW41CXjya+rpQqJ4np/lerkZ5+eDU+/IWvcCCs DpqZzYXloffzm30X08cqO42vew/nJBZzVF5LfTrVXDyEmQXJ4x1CjC+fKc5o31g0ieiSebM9h8qNb DArvViRQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIQ-006oWj-0E; Fri, 24 Nov 2023 10:20:26 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIJ-006oU2-2b for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:20:21 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-407f9d07b41so10816745e9.0 for ; Fri, 24 Nov 2023 02:20:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821215; x=1701426015; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KcQXJEXcZd7GL4PMaiqftkL6zYVU5SF0XIUjWooAdVI=; b=Jqh6NQ4oX3LZliX1O9ZtYQ+rgid4ftfbDs7qNl1HT+ozo+kUY+CTUnj6EYIL0lhOk3 2xCUmkbz5TMnjsxEOmpomyiIgrDBt8etNS7AOE9ANpgiNi9BZdiRzya+V9bCbmEd9l+J JrrZIZz4fPLxbkCL61s9Pw1F1WfhJ/M2tHpyHoT2dsNvr045NsVaud7AV/bp//UsbjS3 XDPHfDHY0//rd6ER+maMY0V/zhPiQjO4XrP5BZDsxdx9Lw6DS1zMGES/wnMdh5J0fHGM wZX315avVP+NoaornLR21PWk3FGDkDgHDFM/t5LYeUnrptb43eJpmxXAGxocjwm52Gin kcXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821215; x=1701426015; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KcQXJEXcZd7GL4PMaiqftkL6zYVU5SF0XIUjWooAdVI=; b=aa9bARzWP3j2HPcIHqXYBEmCGWbmY0LerUkzT2jRrH/eNVax8H7R3BmX+ig9XUcbzV s7mvV3B+Qn2tXyya5G4M+bjBr0hfPb7hRYLXDQggEW16yC9gMjHOx2g6eBRyI7RtTkqC bTk1QawGpgWW4ioeUOJut8i6XdTOwIXoSNtlquWNRcVYnrU5UxBEtQp1ffKkwQdbkHkD J+ghbfar/9i1hOIGk+YsvngbuDWltBdeSbj2/gN9tMukOqtXAzWX5XSqe2R2lOTr+JJU prgm80SJKaL3SG+y1uq2v31kxFy5sNAkn0PVits/OaVK4q7yWJ+IC89BYzSfRvf0PsBE EzQw== X-Gm-Message-State: AOJu0YwRORVJlFMm8UpzvbxQPQsD5NpUOrxuLKC0akrlAVCMZ2A5Q16Z wUxV4/HU7gBO51u85toO0VaayQC9vPB52k/CrwImFx85tdivqy6GgrImN57jxoWC9fK/rXiSHIM XmS5L+4PEVgbOYHsc+Lyob4MRBe9MJAQbYhmRfSrMUCTlM3xywloHyA5ofjhq6Iv6mWqKdx+HUg w= X-Google-Smtp-Source: AGHT+IFJYrPbuBjHTOej8yB6rdU6C/kSU4LQYRI7/2NhcVy5UIgLdj3wQF2ro041SEc2Ll7Beo8aJM5z X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:1e1b:b0:40a:45ea:ebe1 with SMTP id ay27-20020a05600c1e1b00b0040a45eaebe1mr43880wmb.8.1700821215422; Fri, 24 Nov 2023 02:20:15 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:41 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1050; i=ardb@kernel.org; h=from:subject; bh=Q52rjgTf5a2iluAi5473JQmjpRUrpaaPVzn/tdZq3BE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWholGz/t6RRQfkT01uV2N58XxegMrOTdOmX7MTSFV3b eLKqHHpKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABPx+MrwV3DXwsbvHVVW86u0 26O2+Kzu6rPcvkxC7xGnVLnUiYNvGxn+yi72qPGq/JmzN9Xs0PGgOXdMJ1zVOT/7RWZ5t9vrB7s 2MQIA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-42-ardb@google.com> Subject: [PATCH v5 01/39] arm64: kernel: Disable latent_entropy GCC plugin in early C runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022019_848902_0346AEAF X-CRM114-Status: UNSURE ( 9.98 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In subsequent patches, mark portions of the early C code will be marked as __init. Unfortunarely, __init implies __latent_entropy, and this would result in the early C code being instrumented in an unsafe manner. Disable the latent entropy plugin for the early C code. Acked-by: Mark Rutland Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/pi/Makefile | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 4c0ea3cd4ea4..c844a0546d7f 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -3,6 +3,7 @@ KBUILD_CFLAGS := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) -fpie \ -Os -DDISABLE_BRANCH_PROFILING $(DISABLE_STACKLEAK_PLUGIN) \ + $(DISABLE_LATENT_ENTROPY_PLUGIN) \ $(call cc-option,-mbranch-protection=none) \ -I$(srctree)/scripts/dtc/libfdt -fno-stack-protector \ -include $(srctree)/include/linux/hidden.h \ From patchwork Fri Nov 24 10:18:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467438 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 84414C61DF4 for ; Fri, 24 Nov 2023 10:21:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=0S7pQvq2swJP9tRC22B0UDYUryvA2+3B+QZ8xSua2p0=; b=FtchDvbNXL6wWK9CmOcj7RaHHJ dg7Hdk/YjaxkuvtHX+VJldTN+rtQaRR28ODChB4OzmELJTIhIFZ5Acg8MmPXlChfjyy3d89UrMRxr uuFS6qelHtVXqgGoL3CYjd77+h3MckM8VLs964dOEJ3+Ya+N/KJzj14DqNvZcr/UO2oAyB6PeboDi IeeDSUplR9BJxSYyvNoLocICyjIJ6Ptdv3byrVQF6Ozjfpzvnql6xZ7KSZ5AuhLJxMhvm5SNe+hlx 4kfwWT0/YYnkUxcBmcx1ZJAkf+Pbz7wqq7UqzrZ0dGzUOF6obN4h20HjCSIOuMgp9p1dQu7/PelRx aOc0zKrA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIQ-006oX3-23; Fri, 24 Nov 2023 10:20:26 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIL-006oUN-0h for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:20:22 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-332d26dfc5fso901096f8f.3 for ; Fri, 24 Nov 2023 02:20:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821218; x=1701426018; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=c/0kHqi5ZIPX7uCJiko3eeawbTMfWCgSYmmSJYACJmA=; b=SHvcPSe/aDOSzt2333tBe7o39wRXJmC5N4MuVhKOMtDSQqkO/HOzVvr7+hP75D9jg8 VCjwW8+0WYxnJCOzWactm8TvqP90x49OZAcMopwV3UDKFiYrmdt23aviAXLlV98Hx8+p 5N62vrrweAQ4kXSHZG0Q8jTLXe4hmVE11elbv+PYHKa2cjF//5Z/pv2FTyoLgpGbZcJb Wm5MilD+WsGmp4ezLaNjytGYxJfMVsDaIntxBrQv2GmDf2WtYOGVi9h/hWHuztyNX18h X7T/TvHdz3ZBCiVECtTaXK4lEblQDdcCeDOcXsHTcTYFiDQ6osszpbrpSzM1T6HeBh/V 6uyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821218; x=1701426018; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=c/0kHqi5ZIPX7uCJiko3eeawbTMfWCgSYmmSJYACJmA=; b=fmA4IWJ3JkJVwJf8xsueXiNruk3VOPkWctHo1IYjAlzdiPQhuazr0fFArIstqQbVYT a7PduVbclBN/H1JmfCqqekT24GwFhwcN+TfH/GipWPiM7WxCfZKBFh/9wrLoLlk7Smnt 69Wr3jVZuGJvaSJO6XvH7M9BopaC10XtkJtic4gc/XBQZshGVfvt26/igmU5zUo7WYQv 9VQy8guvpeEKnmLWRN40iQp7lu9VjCDucJ2u33bClA4YYppJVZS78fM9XyUlRdybSUQd T1Xutmx3EL4OMGg3e/4AQAYKcLk/RlWoaa11xokbuQ8B999DXCWAdm0plIDNWiWTMTtD snOw== X-Gm-Message-State: AOJu0YxsRnLjkC5LJ5/cKWruyQer8iOnP6tvdoEpQUhfb/ZdZYZtoxrz OuNidGRg5ctCE1tRfTtJ3fASODn1LXhaW3OdAkwxABFuoyoGzlO34apgu2NrPxMsYmILEYuVuAS Ta7kUFAuE5xsvJHNJb5zWXAf2sf/rqmVh/jdwt0wnqlVKBpDv2ZTl/jxQjIUfWZGYOg2cODw997 E= X-Google-Smtp-Source: AGHT+IEAVjWKTcjFA6zjaEDbdfBdOJd6p4Zvg6KDgoekypRfyB5Zz0A7siNugrYYVorCEV0TZCwFfD74 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:452c:0:b0:32f:7d60:d633 with SMTP id j12-20020a5d452c000000b0032f7d60d633mr35069wra.9.1700821217733; Fri, 24 Nov 2023 02:20:17 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:42 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2831; i=ardb@kernel.org; h=from:subject; bh=ZfuLl88gFL9uskT9e1AlkywoKWtb5bgNB+HGXf4gRmk=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhopnB2bnxmE5ssarqKWFrWYUAP4MNEYphwovN7a1Lx Fmb5neUsjCIcTDIiimyCMz++27n6YlStc6zZGHmsDKBDeHiFICJqH5hZPjvELj469dZ2eHJUyJW 8S/eG7vzt37yovrsG39zFUs6QtkZ/qkE7z30yZff+Oa/85zLCl/+PzzveNLyH84T3DzmfhF02cc FAA== X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-43-ardb@google.com> Subject: [PATCH v5 02/39] arm64: mm: Take potential load offset into account when KASLR is off From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022021_262650_FDB1E3A7 X-CRM114-Status: GOOD ( 18.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We enable CONFIG_RELOCATABLE even when CONFIG_RANDOMIZE_BASE is disabled, and this permits the loader (i.e., EFI) to place the kernel anywhere in physical memory as long as the base address is 64k aligned. This means that the 'KASLR' case described in the header that defines the size of the statically allocated page tables could take effect even when CONFIG_RANDMIZE_BASE=n. So check for CONFIG_RELOCATABLE instead. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 27 +++++--------------- 1 file changed, 6 insertions(+), 21 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 85d26143faa5..83ddb14b95a5 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -37,27 +37,12 @@ /* - * If KASLR is enabled, then an offset K is added to the kernel address - * space. The bottom 21 bits of this offset are zero to guarantee 2MB - * alignment for PA and VA. - * - * For each pagetable level of the swapper, we know that the shift will - * be larger than 21 (for the 4KB granule case we use section maps thus - * the smallest shift is actually 30) thus there is the possibility that - * KASLR can increase the number of pagetable entries by 1, so we make - * room for this extra entry. - * - * Note KASLR cannot increase the number of required entries for a level - * by more than one because it increments both the virtual start and end - * addresses equally (the extra entry comes from the case where the end - * address is just pushed over a boundary and the start address isn't). + * A relocatable kernel may execute from an address that differs from the one at + * which it was linked. In the worst case, its runtime placement may intersect + * with two adjacent PGDIR entries, which means that an additional page table + * may be needed at each subordinate level. */ - -#ifdef CONFIG_RANDOMIZE_BASE -#define EARLY_KASLR (1) -#else -#define EARLY_KASLR (0) -#endif +#define EXTRA_PAGE __is_defined(CONFIG_RELOCATABLE) #define SPAN_NR_ENTRIES(vstart, vend, shift) \ ((((vend) - 1) >> (shift)) - ((vstart) >> (shift)) + 1) @@ -83,7 +68,7 @@ + EARLY_PGDS((vstart), (vend), add) /* each PGDIR needs a next level page table */ \ + EARLY_PUDS((vstart), (vend), add) /* each PUD needs a next level page table */ \ + EARLY_PMDS((vstart), (vend), add)) /* each PMD needs a next level page table */ -#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end, EARLY_KASLR)) +#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE)) /* the initial ID map may need two extra pages if it needs to be extended */ #if VA_BITS < 48 From patchwork Fri Nov 24 10:18:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467437 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 530EBC61DF4 for ; Fri, 24 Nov 2023 10:20:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=HciSN5V7fEYdUB/CO/XmpIMTmSc6U/ODnM+0MalWNeE=; b=PlHks/Fx8Gnbo4rJKeFjoQe6hG XftWLE5HOKxNV+At8oE3o2DrhMwA4l9VFPZMCf20uPYomBvEHqKQUywF3ESrVkIpge14tFdTCeCv9 dOwM5u3Djs/UIcGt1pdWoX+9x8fKnlyRsQpP3fGr8tbewHsqmT5D03Sx6x/dBHD+1MdOLtJszj+4i 6EVuIvveSvOQNhjvZSfvWk16PeJnOLaZSS43/Fq7o5ZtySXm5Id9QlyJOaZXC0yD22r7vOActuIwH OZlxhd+c5pzlAZ2KKAw3lAMoXcQdDj3UWb/oRFyHY5OnS6OcpJEe+aJBCeKVuPq7MShUs8Cqgjc4m 3m9yUU7Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIR-006oXO-0y; Fri, 24 Nov 2023 10:20:27 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIN-006oVA-10 for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:20:24 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-daed1e15daeso2207503276.1 for ; Fri, 24 Nov 2023 02:20:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821220; x=1701426020; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=vLytofFJeC+aEU9SPZ239oAWyj2r+ByEUn7mC6VxtZE=; b=bGc3IAmtLHB85aDrkzSK3E96rDwn5UC4UI62U14yitpNnlt5aKhj5xo+kFisVgwyn3 HhDlbYql1lVlpIrvBCw+cS5wM4iauj9EW63CCfGlqQfR+IALDeNq52YLI/ep0YGjopxz cd8dzqWZ97J0n5dbDiAWvzChSDzOx/9oU9ni4fRXu0rnutrxZxA1SvC6EGEWw9M0HTtK ikyUppHP+6uKO2Jcw9mgBzmThDppZGceLtyEo4lVXdz11PAXCFmIXomiAsBkCisytKCQ 0j9RusbDX0EfKL5xUUfrDQnWQ/lEiEDZOzHC2NtD/2NjaaeLq1ispwapegp4cdGZ30St y1Rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821220; x=1701426020; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vLytofFJeC+aEU9SPZ239oAWyj2r+ByEUn7mC6VxtZE=; b=sDEQKgsSF06ZL135ySY6ljaA5ytVQiSOd7DybqZ6LK/vqfRrvVbM5alpUqpVFtLz4T Rq/VI4qSPsSYMKRY8xZbvgsFINr9UwrqW+Fy9ZQIKeLnw16o6AbXyi+nPumVCihK66Kg F2DUdE8NoBxCu7eQfkpdNgVLvIQXeFpSXcLnbfdaYTd5apbOnBPAPUJlrXImj4bEt69W YmlfIp0qYxH3xLayEc4vDtoT6NFVyOMiCCFwhcPJzr5tAeqWqOhXFn2N0xIJVJJbRVsV PeRwu3Ow4XLIKskkIiAqVEs0QPTo7ojSqlpZLoDUmTvaYMGImv2iSUznHpKW+2vuzgRm PzkQ== X-Gm-Message-State: AOJu0YxDZxeIyLmfPfuSqzoYk8zUkDouAws5akThm04TI2s2BszBHSiF LfOenQvA73+SlD/hRJ1JkiwK3XB5T94rsK43hN292R16dWYgxaWrFLZdj6yNnlp8ks0quw4QczL Ktuq39d4ywl1sbGimPbZjHhazO4c6UWDFwcVK5jVf/SvQcfe3BZeTIg2eYXF37mSDHsp5xoJ2In M= X-Google-Smtp-Source: AGHT+IFaVekewND1KIaYF14cMvl9F5vIq6DqKcV3+5z4JjEkATYJTBnRBjMdsjhv06kQ0rQoKqkxmqnA X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5b:f0b:0:b0:db4:5eed:8907 with SMTP id x11-20020a5b0f0b000000b00db45eed8907mr52424ybr.8.1700821220398; Fri, 24 Nov 2023 02:20:20 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:43 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2455; i=ardb@kernel.org; h=from:subject; bh=y2jrY5S1I/ojQOOhnDBpbwtTbesmY9zlrLnpWZZwxOg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhokVPhXdrokZKTl4r0xLPYP2TR/oyHLNrZ+u9CvqWt 6DhtWBHKQuDGAeDrJgii8Dsv+92np4oVes8SxZmDisTyBAGLk4BmEjGAUaGbZNnBV3pLLo9Y+vu I1bhun8fTz7Sff5AaW+brHzg3EiNy4wMv96dT/+/YeYltvx458bERI5Hs15xXr8cnftS0tWYwe4 hLwA= X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-44-ardb@google.com> Subject: [PATCH v5 03/39] arm64: mm: get rid of kimage_vaddr global variable From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022023_349226_7E9536E7 X-CRM114-Status: GOOD ( 15.05 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We store the address of _text in kimage_vaddr, but since commit 09e3c22a86f6889d ("arm64: Use a variable to store non-global mappings decision"), we no longer reference this variable from modules so we no longer need to export it. In fact, we don't need it at all so let's just get rid of it. Acked-by: Mark Rutland Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/memory.h | 6 ++---- arch/arm64/kernel/head.S | 2 +- arch/arm64/mm/mmu.c | 3 --- 3 files changed, 3 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index fde4186cc387..b8d726f951ae 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -182,6 +182,7 @@ #include #include #include +#include #if VA_BITS > 48 extern u64 vabits_actual; @@ -193,15 +194,12 @@ extern s64 memstart_addr; /* PHYS_OFFSET - the physical address of the start of memory. */ #define PHYS_OFFSET ({ VM_BUG_ON(memstart_addr & 1); memstart_addr; }) -/* the virtual base of the kernel image */ -extern u64 kimage_vaddr; - /* the offset between the kernel virtual and physical mappings */ extern u64 kimage_voffset; static inline unsigned long kaslr_offset(void) { - return kimage_vaddr - KIMAGE_VADDR; + return (u64)&_text - KIMAGE_VADDR; } #ifdef CONFIG_RANDOMIZE_BASE diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 7b236994f0e1..cab7f91949d8 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -482,7 +482,7 @@ SYM_FUNC_START_LOCAL(__primary_switched) str_l x21, __fdt_pointer, x5 // Save FDT pointer - ldr_l x4, kimage_vaddr // Save the offset between + adrp x4, _text // Save the offset between sub x4, x4, x0 // the kernel virtual and str_l x4, kimage_voffset, x5 // physical mappings diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 15f6347d23b6..03c73e9197ac 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -52,9 +52,6 @@ u64 vabits_actual __ro_after_init = VA_BITS_MIN; EXPORT_SYMBOL(vabits_actual); #endif -u64 kimage_vaddr __ro_after_init = (u64)&_text; -EXPORT_SYMBOL(kimage_vaddr); - u64 kimage_voffset __ro_after_init; EXPORT_SYMBOL(kimage_voffset); From patchwork Fri Nov 24 10:18:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467440 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3E551C61D97 for ; Fri, 24 Nov 2023 10:21:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=hXCXS/r7qe6ZCmeisWY0SmRpmZ3aN2oujPjs/0uiKJ0=; b=TQo3VPYE/3/zvRq17XHC3V41qq sM4ChoVZs1yymlooj92bN71prsHzGvQo5ZDwZNgiGQlLnXgN4v0WdWQ87kzSryzz4GbRErPdTsdu1 MXszlzgQwKTjy2pTvsbO5Txb4DKSTbc0zYMTKXRJOEl8z5BNbmvLyuGIWxkwxxmFgIx0BpQ/HPzNj +Pa7Kf6zo4v13XBgUnEdxylxigk9dG9CPjjF9zHWt+fS93u517KJPqrHdSfZ+S8TwpbBC0qP/b0Wb sgjICZYf4ff0YiAX8szUV9GIhZnNQol1/CmyBXpEcgMfFIP2xO0F7T+xnOKFjNFZoYren5Av2xzQZ KfxSInRA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIh-006odf-1Y; Fri, 24 Nov 2023 10:20:43 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIP-006oVq-2r for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:20:27 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-40845fe2d1cso9635035e9.0 for ; Fri, 24 Nov 2023 02:20:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821222; x=1701426022; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GLTAw6Fzy09giJTA8KTIyyYCeJNNJvn+fc4BLUgm6z0=; b=lpdOjXK1lxY3UuaP1Js++VTZPeai2OZlC0a9IpgajaA55mgyg+y6Gn0Q9s5k5lp/uG ZvlAaQyf2B3J9nvlVuMYi4QQw8RW8dEuJc1P+mevFgvOSjeEHGXGNIKh6Co4iF7pcvTw kdaOC9SURZ0dXnb7A7FwqaxLE8jMDRM072jJS/VrEdBURUQzSfEji8aLt2IPv7vLLjnK Wj84sl/aWtASGV1ufJkgbQtMFcSuzV2oyT/9fcrNQXidycYCh1N+JAoitnfq2sBNEiQ4 fKPmMWOZtuDpzJ7ydz1MVPbplsG9FnKR3p4n5jUJBhW2ThgL2H/JCAuny5V+bcm5fSpW E0pA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821222; x=1701426022; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GLTAw6Fzy09giJTA8KTIyyYCeJNNJvn+fc4BLUgm6z0=; b=ObnhD+E5BVc+DMngMeuuSi/JAPcLVSNUXwEkSqsAUHCi/t7PElZAR4FS3kxfnEPHVz hjfFzQjOXZLtBObJjakJmzQ6vDiOPQpz1V05bqqie1obiUqLc0upsDWyQuAFQv0ZmpyB NPyOCTvsxuqnIPq7FtY5enG4VmRl+x4KpwXHydI/xlnY7raMa0p3UWNuSIx9KpvT45RO w8ODAUxfGd1hTvF8CnwngjPSvUnayiwfAfMWaXHsfugwnKG9BXiOSGRf413XPlBXoNNr Pn1X2t0FElfXmaB4ki/yT6bbRbKRZPRN2MtdX9B3t2Dq2/6V2t2QAzVEQEvNEWSgA+rf RlWg== X-Gm-Message-State: AOJu0Yyw3h/W1i1uEdnbcsbnWR9B09C3KHaq/G5qxpT5lHP6hFM9cNsS kYVzkhU8SS3P7J8dqZrv4gAr2nTerBxN03xhKTmPlQ48nczoIlGz9Mey1UCetRQ0Cu63VxDAtUv czHJX4BYIIvse9oKrCcZ0f0VwX/pTYhMHQ9zpEFv0ZTaKhEc1v0myso/uXpVBS0/4bsrsOJ+7qG c= X-Google-Smtp-Source: AGHT+IEczOUfDAmOsmeNdwhqK6Ae3oUN/v15v8AKU/ZtD66TkBwof0b9d4m/lCjZ2Rlu2TFTNwEr2yR4 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:3b14:b0:40a:4554:dd1e with SMTP id m20-20020a05600c3b1400b0040a4554dd1emr45553wms.5.1700821222698; Fri, 24 Nov 2023 02:20:22 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:44 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1735; i=ardb@kernel.org; h=from:subject; bh=jbMXLbySNrhkt48hluR0iEseEJX6j5kE6/EXn3ZHfgg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhovXQ7brjx/auEZ3CtfzUhog/m+4w3HM66+NovfiOx MInj286dJSyMIhxMMiKKbIIzP77bufpiVK1zrNkYeawMoEMYeDiFICJ6M9l+Cu8cabo27fTVHmM rpyf7LPJ7MRy4ZlHlz0onpBSqlvcVyPD8Fdyj/tzm26ulVx93Tvaf4j+/v/z3/ct666WhCjvdOA UecsDAA== X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-45-ardb@google.com> Subject: [PATCH v5 04/39] arm64: mm: Move PCI I/O emulation region above the vmemmap region From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022025_923700_046ED2C1 X-CRM114-Status: GOOD ( 12.18 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Move the PCI I/O region above the vmemmap region in the kernel's VA space. This will permit us to reclaim the lower part of the vmemmap region for vmalloc/vmap allocations when running a 52-bit VA capable build on a 48-bit VA capable system. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/memory.h | 4 ++-- arch/arm64/mm/ptdump.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index b8d726f951ae..99caeff78e1a 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -49,8 +49,8 @@ #define MODULES_VSIZE (SZ_2G) #define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT))) #define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) -#define PCI_IO_END (VMEMMAP_START - SZ_8M) -#define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE) +#define PCI_IO_START (VMEMMAP_END + SZ_8M) +#define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE) #define FIXADDR_TOP (VMEMMAP_START - SZ_32M) #if VA_BITS > 48 diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index e305b6593c4e..d1df56d44f8a 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -47,10 +47,10 @@ static struct addr_marker address_markers[] = { { VMALLOC_END, "vmalloc() end" }, { FIXADDR_TOT_START, "Fixmap start" }, { FIXADDR_TOP, "Fixmap end" }, - { PCI_IO_START, "PCI I/O start" }, - { PCI_IO_END, "PCI I/O end" }, { VMEMMAP_START, "vmemmap start" }, { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, + { PCI_IO_START, "PCI I/O start" }, + { PCI_IO_END, "PCI I/O end" }, { -1, NULL }, }; From patchwork Fri Nov 24 10:18:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467442 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5BE1FC61D97 for ; Fri, 24 Nov 2023 10:21:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=i4uFqrh7mJUTEvAXP5HoQD/OA61eAV5WHKSAlEEtRYo=; b=b0oToE5JNYI2x+eZi9xUJG5z7H Bw8RJtnyQWlLXR1huWrPD11QNG/eCLPuxtGPW8hueK6Bmm1nXbs7WyodbUY4x9seRaRZSCIChlmS3 xGlrVz3v+7QOB6E1x7hyEcb1Of5mOzMvwXJqCSyohv+hmrtZRrVivHonYBGJlWMyTm0E7UriEC4KQ UpWz8Q1XGvSbehHyf3KBneCyAAXs044G6uBsTm4Rdf7xCc4iPgC+Ch/Yi8l8gwLfV4IvxC3HN2YjH v+Ymzqu7GOn1W5bw5kBWu/yvn703x8NB79j1HKf3iceoidRXhPaUqL5eZJ9xl0nDZ7bEXJm2bKuqQ L62E6owQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIi-006oeG-0S; Fri, 24 Nov 2023 10:20:44 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIR-006oWN-2B for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:20:29 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-da03ef6fc30so1890568276.0 for ; Fri, 24 Nov 2023 02:20:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821225; x=1701426025; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=t56SIbSQ4jhUIaRNPn2NUOn72ywmn1puZUeTkaD6WSo=; b=RuXO/aj4O+rzKXUd4GmD9A75SC4bQTZxwAoCl5jXR/hflijD1dDZ+EjRiDGWYoKD1o OWB9vmkxGa8co2q1czDvzubcsdWiZlAypTwvK98bYHU9MgZLDWvrHfQ6ApOVKMCAgeiE bAt0WINYnH1zncf3Awb6qRcFFIo4TE0zuXNHPOkEyePO+q7zYk6tsw8sxsqWu//JAeSc PgyD30J7rszJb+WtdNAHkOqHCYuwEaas083iGKdUWhzqXbRvfGDeftCeohCOC1cfT11/ QPiGbzLyh59P5/oOtgOY3BGu4dR4ScVNQgefFIDZ0HhdWKmm+lOhBvLW3HYG56YaGn3n UYBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821225; x=1701426025; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=t56SIbSQ4jhUIaRNPn2NUOn72ywmn1puZUeTkaD6WSo=; b=uIGire/XHAtt4WIxnmEfj1Tdo50n6WLXxWvAqZcWvcKuqE32jwA1/MlqBOJ7nJE2Kd HRLOIwVmwpzN5seGdCPEiwyzf05sq8n45BZuSen/mB5ankwRX5R914o0P0z7Z52uv009 c2ZWbneGC5yynVt26I4uj7ES+WwTt4Yti6YB6hiiNcCAEcmQZf6z/0ODPOwT7u6sN8zZ AP4NMux4iilf3RVMmTaSxhBeHdRL5eYlrH64uzNGpqtuFpff9ypfIOCDt8/uUagetoUZ Z4MVzkiiYN88s29GczspF0lomBTVGcieyTtBte9FHJOweOcSdriZ55wUl/ok+wFD2RZ9 zTjQ== X-Gm-Message-State: AOJu0YxWe7Ksua8zN8tMfMEjZyQh+XHNopTKebxaDlQHXz9U1mzRJlEB gvVx0sGzJ8BgSUXyZQFMI9AdxgDrALf6ATSVWuKtipn7Es7umIjE+E7ysPx7TPM3uz4Xf/RPUrF dvIEKkaWPJNxyjQYhI+5A2/dV/ke/603f7fQMOw72wT6iLFPbsXXI6YBkuVWmQwTFsoYYjEyROE Y= X-Google-Smtp-Source: AGHT+IHtdZXMKUdmFc+sAaEkq3R39NRqcq+N/nbFxsq+JG7XpncajNcdqJGKRXddNnrqVfkCrdLFshha X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:6f8b:0:b0:db3:f436:5714 with SMTP id k133-20020a256f8b000000b00db3f4365714mr81880ybc.0.1700821224902; Fri, 24 Nov 2023 02:20:24 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:45 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2528; i=ardb@kernel.org; h=from:subject; bh=VDbck9uGRLHEcTODagyQXSJqFVu7bfeIZMFY24kXmn0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhom3p3VWyuzZVrrXKYenSsg+5lvhCOvmznNuGbXt3X JPkXLauo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzkkQ7D/4DdrbpLdx50tJvd KFTkdFJqhfb+v5VFk+t/G05aaPfy+1JGhhul+9emhu6UUXzc/Ka3tCDM8q7b6Q2OzxfN/WOTF7L xBB8A X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-46-ardb@google.com> Subject: [PATCH v5 05/39] arm64: mm: Move fixmap region above vmemmap region From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022027_718490_740B24EC X-CRM114-Status: GOOD ( 14.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Move the fixmap region above the vmemmap region, so that the start of the vmemmap delineates the end of the region available for vmalloc and vmap allocations and the randomized placement of the kernel and modules. In a subsequent patch, we will take advantage of this to reclaim most of the vmemmap area when running a 52-bit VA capable build with 52-bit virtual addressing disabled at runtime. Note that the existing guard region of 256 MiB covers the fixmap and PCI I/O regions as well, so we can reduce it 8 MiB, which is what we use in other places too. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/memory.h | 2 +- arch/arm64/include/asm/pgtable.h | 2 +- arch/arm64/mm/ptdump.c | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 99caeff78e1a..2745bed8ae5b 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -51,7 +51,7 @@ #define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) #define PCI_IO_START (VMEMMAP_END + SZ_8M) #define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE) -#define FIXADDR_TOP (VMEMMAP_START - SZ_32M) +#define FIXADDR_TOP (-UL(SZ_8M)) #if VA_BITS > 48 #define VA_BITS_MIN (48) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index b19a8aee684c..8d30e2787b1f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -22,7 +22,7 @@ * and fixed mappings */ #define VMALLOC_START (MODULES_END) -#define VMALLOC_END (VMEMMAP_START - SZ_256M) +#define VMALLOC_END (VMEMMAP_START - SZ_8M) #define vmemmap ((struct page *)VMEMMAP_START - (memstart_addr >> PAGE_SHIFT)) diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index d1df56d44f8a..3958b008f908 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -45,12 +45,12 @@ static struct addr_marker address_markers[] = { { MODULES_END, "Modules end" }, { VMALLOC_START, "vmalloc() area" }, { VMALLOC_END, "vmalloc() end" }, - { FIXADDR_TOT_START, "Fixmap start" }, - { FIXADDR_TOP, "Fixmap end" }, { VMEMMAP_START, "vmemmap start" }, { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, { PCI_IO_START, "PCI I/O start" }, { PCI_IO_END, "PCI I/O end" }, + { FIXADDR_TOT_START, "Fixmap start" }, + { FIXADDR_TOP, "Fixmap end" }, { -1, NULL }, }; From patchwork Fri Nov 24 10:18:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467439 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A2AE8C61DF4 for ; Fri, 24 Nov 2023 10:21:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=mpdOE88F4X+oEZoFLweSRZgtbt9EBm4vf38NE5eREv4=; b=IPRsSU7ijKUKBO0NZnRKDumXCA B26NTTiuWwdYNUOrDhuLHnXib9R4tHxwg1wmT5isYgJQR2m5nxGyz8jVGclS8NyT2N6wPPb+sJjPK s0EpcozN1EdH8wOFwJILAwe4Jx7G12KQneQ2bcfLmyETibgNvtZVNygEIEAfGtYn0FE4FoyR0VVPE POJpH0NNbjylspu58WRL6zGeMku15c1ellhSkeZPd4Yt7BvtDLMARxMd9lf3VKZMxLU+6UUnmV31i Swvaol6JxNDoJsDBDYAoxpYjcGbxk+XumYD8aDHgcVx//GdM1Yntgx7v+vuViI0twc3QFfIKLd5uq xESsvhWw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIj-006oes-0B; Fri, 24 Nov 2023 10:20:45 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIS-006oY0-1e for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:20:30 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-daf702bde7eso1718058276.3 for ; Fri, 24 Nov 2023 02:20:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821227; x=1701426027; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QcyQzQvrhZSKuGh+IrENHro9WhaIhDqkoV92ZoY5+Bs=; b=co9jQiLzyAyVNvv4KrNxfE5u4nWQbaPmkFWkgTXG+UYgclye81u5mKk0zdg7efViMn JB0HAFDE7X4aoNt1M5Bzi0dozaOjKUhTuSJYxR0ssAS78t8+Zq4SfeG7ePS+1zdbyIa8 vM26JU9BtLL4LdYmiLDz/AbHzZ2XD/iT/BoJYDFai55qkJ29X8CIBdSYXZyug9IRfOUI JS5s/EhuDwETnOBOPnelM7bp7M/ABGqJUt5at/PYyPwOQtaUvJi35elvdiNK4A5EUgs7 RJaWp7DHR4sv8t3rJHVhNRxtbXPVOeb/RT2pTp+mLtHf0Ggd9m080kCCabQBdluQeESe q7nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821227; x=1701426027; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QcyQzQvrhZSKuGh+IrENHro9WhaIhDqkoV92ZoY5+Bs=; b=cXGofH372oZ7Z+zi7zIjqBmCJGF4uXvvNidO/89n712NCUKo4RRhJ0vAhAyEW1EQOj vi8MULB66NogMH1avkntvgvDcBKf58EKECWvQsu15I1G7k8Hmn4+mvAKYPjLRs6VyKfX AjOD16Hfiy/uzC1ySlnFS2NGBUQRCvNyoqNyB7UYGoW8p7I5JAbE/7HOrMbhn8np4gyW xDU+A/PNgjtXZ1HmU5TmFFIHm6gg/9S7gWNvXZuPQTXAME5EkgG2ZfCcF4/C5/JVxzI7 BLL3FhjF/fU+KGjmHU4HP8Elv+leJnnyZ5iQt9LJH2c+AyYNS+HVX17ynf7oEe0qnPTL xQ1w== X-Gm-Message-State: AOJu0YwJ2md9XnKbRYkcx1CpWZyevI379DT0lxC2HsXmVHOBIpknlu+Z gLKYlNKs6N0lZCWMD+XosHSvBla2GIOetP+QIjl/nYAGj17S7ycyzy8NgGCJLrxcaK/VgR7XsS1 IZtSjCCHaRGViCKFxz3/9XhOT6FQuiAAwhF44lBiqIGl+17GWruyXephRim6i3HER+Td7Jb63Vh 8= X-Google-Smtp-Source: AGHT+IHyU9REEENips3T7qy8T/g1qUI6MTSIlvCldPGNAWqwDeKE++E6HENy7vQoiYqJuhBUO/Ij7oRy X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:7141:0:b0:da0:c9a5:b529 with SMTP id m62-20020a257141000000b00da0c9a5b529mr49857ybc.12.1700821227216; Fri, 24 Nov 2023 02:20:27 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:46 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3329; i=ardb@kernel.org; h=from:subject; bh=WFd4jQ912fgRJirpPKRfxk61zUoZ4kmCkhvh5bpu8Rc=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhov2I4/rrlxdn9EnHb9FZXsdouEtLdf3b+GM9l/7tN KxSsVrdUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACYy/xYjwy+uS85nj1/iecIY FKb3UPOskXy6+Ia8msPRLTmTtZPadzL897NJu7Swn0+0X/qh2uaHM1K+OV96cWtJTNmPJWf4zkw s4QAA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-47-ardb@google.com> Subject: [PATCH v5 06/39] arm64: ptdump: Allow all region boundaries to be defined at boot time From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022028_549383_59210C6A X-CRM114-Status: GOOD ( 14.02 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Rework the way the address_markers array is populated so that we can tolerate values that are not compile time constants generally, rather than keeping track manually of the array indexes in question, and poking new values into them manually. This will be needed for VMALLOC_END, which will cease to be a compile time constant after a subsequent patch. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/ptdump.c | 54 ++++++++------------ 1 file changed, 22 insertions(+), 32 deletions(-) diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 3958b008f908..bfc307890344 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -26,34 +26,6 @@ #include -enum address_markers_idx { - PAGE_OFFSET_NR = 0, - PAGE_END_NR, -#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) - KASAN_START_NR, -#endif -}; - -static struct addr_marker address_markers[] = { - { PAGE_OFFSET, "Linear Mapping start" }, - { 0 /* PAGE_END */, "Linear Mapping end" }, -#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) - { 0 /* KASAN_SHADOW_START */, "Kasan shadow start" }, - { KASAN_SHADOW_END, "Kasan shadow end" }, -#endif - { MODULES_VADDR, "Modules start" }, - { MODULES_END, "Modules end" }, - { VMALLOC_START, "vmalloc() area" }, - { VMALLOC_END, "vmalloc() end" }, - { VMEMMAP_START, "vmemmap start" }, - { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, - { PCI_IO_START, "PCI I/O start" }, - { PCI_IO_END, "PCI I/O end" }, - { FIXADDR_TOT_START, "Fixmap start" }, - { FIXADDR_TOP, "Fixmap end" }, - { -1, NULL }, -}; - #define pt_dump_seq_printf(m, fmt, args...) \ ({ \ if (m) \ @@ -339,9 +311,8 @@ static void __init ptdump_initialize(void) pg_level[i].mask |= pg_level[i].bits[j].mask; } -static struct ptdump_info kernel_ptdump_info = { +static struct ptdump_info kernel_ptdump_info __ro_after_init = { .mm = &init_mm, - .markers = address_markers, .base_addr = PAGE_OFFSET, }; @@ -375,10 +346,29 @@ void ptdump_check_wx(void) static int __init ptdump_init(void) { - address_markers[PAGE_END_NR].start_address = PAGE_END; + struct addr_marker m[] = { + { PAGE_OFFSET, "Linear Mapping start" }, + { PAGE_END, "Linear Mapping end" }, #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) - address_markers[KASAN_START_NR].start_address = KASAN_SHADOW_START; + { KASAN_SHADOW_START, "Kasan shadow start" }, + { KASAN_SHADOW_END, "Kasan shadow end" }, #endif + { MODULES_VADDR, "Modules start" }, + { MODULES_END, "Modules end" }, + { VMALLOC_START, "vmalloc() area" }, + { VMALLOC_END, "vmalloc() end" }, + { VMEMMAP_START, "vmemmap start" }, + { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, + { PCI_IO_START, "PCI I/O start" }, + { PCI_IO_END, "PCI I/O end" }, + { FIXADDR_TOT_START, "Fixmap start" }, + { FIXADDR_TOP, "Fixmap end" }, + { -1, NULL }, + }; + static struct addr_marker address_markers[ARRAY_SIZE(m)] __ro_after_init; + + kernel_ptdump_info.markers = memcpy(address_markers, m, sizeof(m)); + ptdump_initialize(); ptdump_debugfs_register(&kernel_ptdump_info, "kernel_page_tables"); return 0; From patchwork Fri Nov 24 10:18:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7F28AC61DF4 for ; Fri, 24 Nov 2023 10:21:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ZtVPOHZAkMsHLKLcLSOImxTswlPrpXMruBrrWjWInNw=; b=JDExdvYE/rd8DuoJc5jkz9XHnW QLDyYU32gUqJ1du7J/M+QWuYB+PFs9jt3HwiItvO9trYJLQzPDiqRCmPZ/oVk/yLMsu1m5xbt+Zy2 gSzRtec9B3Hejz17aNA6cIkrQFj1lZemCWJyN01EVw3NV8BxpAXy0PBpq5HpIceXeX9Lhob4ZsL6N nZxVppYahf5gUQvjI9iyJG3xDXlqf9RhisHwS8Vqx62sV2J2J2ERRQb+e7eTp5yiRoopB+bZTv98R AJ10BwpfqzB1NCz1ArCqcokitegaS6B4ifCv4C+cOA83KEpkkggoHrS96znU8HqEVBWW5T9ihqzJG 5KJb9nTw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIj-006ofQ-30; Fri, 24 Nov 2023 10:20:45 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIV-006oYs-0E for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:20:32 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-40b38501d82so5660425e9.3 for ; Fri, 24 Nov 2023 02:20:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821229; x=1701426029; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5U8g5+sbQRRfUFlv0vh5r16aYK6GyXSTv5pkFBJ/hu0=; b=JKpqPh2LQPEihZac9xbjdvcnFQ7SyFTf3tLnrdDhma1W45uTlExGW7vNcVN4jb++Hq 4U6Emm0SIOJU0Jc5qji+bR7BQi0EKCH2cf7fA+vtix36ikZi1Osls3aNMvw4wPk+fFS5 6S0XM6tfqpIDqSd3jGS/PjI2fBbPjls5ypxkCcD/BMazF7/hMzzczU/aFrkumo74nA5D 0N8EfUS5V1u5nCAv8WZhsc6nWr9OKI2zFd0RW5aAM+YKlY7b6WBNmjKmJ2qZp2wjTNga uE8kGjfDH7ZBl5kXKWbcH5VDZ8625eZ8+ppnMAURnBxijX5oRtcbbrZS9/p/2yQoH9cB SNiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821229; x=1701426029; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5U8g5+sbQRRfUFlv0vh5r16aYK6GyXSTv5pkFBJ/hu0=; b=gQq4hPrTU4rakoySrwzdur8NbeUwaA5EBgzKpIjH+hSIxYRb3p8CdSG03ryY1fGjqM Exbgj5inwaKpWSYbnVQQ9XuMy4nU2oIJQTUjnXffuORk0CVhHpUmMrZBdogPc27DBcAy MwmSVuePv6MQ/I2Z6pIGfxhzbUwBr5oi2hD+IuAMzoUroqHfSwUWfL5y3zumonKjINGr qpSf3o0LWf9eFZlZpKPv27b8sCKUyhRTpk7Spi/xTucLopSF51mRvHvQ5u5u0s5acsPo e1mKW74qTJWAd9/D2/Wb7Ob5AXXC2d20TjGHApysKYyShHvM6fGc6oejq0hg9fMJV8j4 vtVg== X-Gm-Message-State: AOJu0YwDEIDGYtREoOt5HMlmKzZo4KjZMFrbfJi+kFhevVPk4zNHch0y /Pd6qn60xwjrZZoTJ+7bzt/ORemsdLhMYdWRHUXJ4EBfQ0TNdZhtv5t8WyR6dzcg0klL3omGB1A cdzm4Jtdr71PIAOmntgWXPsIO3bR9yi+ded2gXClzi7JukC/8YFLeH0165dHSejiujqqHJvenGM c= X-Google-Smtp-Source: AGHT+IE9ZUHJ5zQ21IXaeWuxsY77WCfQ/q1sob5EH4NQW5qqhefRyeeKiGj6MLf569nq97YeK1i2J7/m X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:5403:b0:408:3692:3d10 with SMTP id he3-20020a05600c540300b0040836923d10mr44984wmb.5.1700821229333; Fri, 24 Nov 2023 02:20:29 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:47 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1395; i=ardb@kernel.org; h=from:subject; bh=QidaWV6xZDE4CF28OiUCmJHc8xL0G7mC+yBlF4AG0+o=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhomNSko3b7pTbJ+9H8pXaMN7dqSDp6n/uYv1K23Oc+ 7YnvWvqKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABOR8WNk+CwWaGNzw+HSPLXN 3M8P5pQc36UrGx5p7cXwpHb5qb3C0owM74qYD0TXzDmZ/7v9mSzrjV/vdxzZ/PjN1udRvte5Dy4 6wwAA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-48-ardb@google.com> Subject: [PATCH v5 07/39] arm64: ptdump: Discover start of vmemmap region at runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022031_121012_E2CD76CB X-CRM114-Status: GOOD ( 13.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We will soon reclaim the part of the vmemmap region that covers VA space that is not addressable by the hardware. To avoid confusion, ensure that the 'vmemmap start' marker points at the start of the region that is actually being used for the struct page array, rather than the start of the region we set aside for it at build time. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/ptdump.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index bfc307890344..f3fdbf3bb6ad 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -346,6 +346,8 @@ void ptdump_check_wx(void) static int __init ptdump_init(void) { + u64 page_offset = _PAGE_OFFSET(vabits_actual); + u64 vmemmap_start = (u64)virt_to_page((void *)page_offset); struct addr_marker m[] = { { PAGE_OFFSET, "Linear Mapping start" }, { PAGE_END, "Linear Mapping end" }, @@ -357,7 +359,7 @@ static int __init ptdump_init(void) { MODULES_END, "Modules end" }, { VMALLOC_START, "vmalloc() area" }, { VMALLOC_END, "vmalloc() end" }, - { VMEMMAP_START, "vmemmap start" }, + { vmemmap_start, "vmemmap start" }, { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, { PCI_IO_START, "PCI I/O start" }, { PCI_IO_END, "PCI I/O end" }, From patchwork Fri Nov 24 10:18:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8DB29C61DF4 for ; Fri, 24 Nov 2023 10:21:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=redvmOFStJpH8SJ+XRx+tMhtWl1PvUCUhqfgquxxEJA=; b=2vPQPu8/FNSo253VCapvswWjL3 5A3fpAnfYjBkm07KIDLhau7OHuJYfQ+zPMhqiZWXGYNcdfUmbC6jt6WRgiaQEHcr3eEFUDv/JO4sM CfQcaskw5LuOFKAvwAWc1au1C+b/oRp7ic5nYGzUO6TV/d9Zp6xDvEGTIZ43op6PjTXRj1Xym5IAx tYdNLgZTqREXSWKjEN+3Z/ak6ZFLO5UIEBE8jCvMwjTxIQqXAKAk3hxpe9AYSLQRbpyFYYxxOKNUX Oa/iup11dQ+4xI0nKLDXqzWDtMJSljg6gNQHLk2aZg5d2s53Uhrz7kaJ3fmdD7kjtd6PAOT+yWhH+ p9QRfl8Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIl-006ogU-0I; Fri, 24 Nov 2023 10:20:47 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIX-006oZf-1b for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:20:34 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-408695c377dso9358495e9.2 for ; Fri, 24 Nov 2023 02:20:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821231; x=1701426031; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=V86olf1EKJaGQIDLZDyTyPOGst5YdmjimBytLucyYgQ=; b=1d+suDUtmr/YWzgHRXaAQM0WLTclzP2MrdZWW24pcdU34p7bTe76ckPmfUcZ+yrlBU KGSIIyFtF3yoNJiGFcPCVnxNqpfMKExiRo+AT1Ds4Ry7Pp3zT0yzhqjZ5j2CCI2Ira1C rOQNoJRoc6p7krhNOxpgIAReaPwukqa7iRzRkLg4Unxa6pyvYSvTZy5vh2ksYxhCaMJx dLC94heYWdp+Orf2jWurCuFANKUIDWsXoDVpFdMLo+agLorm1AL5+3S6Yx81+sGVGpMR B54j1OQKcqWj6yD5naNeMj28vJLOo4CyTQd6DPjqjL4afXi0SIcikI9ww6HBkrQFXQef E24A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821231; x=1701426031; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=V86olf1EKJaGQIDLZDyTyPOGst5YdmjimBytLucyYgQ=; b=DrpSc5L/hPcdbu3XjZEXqWRhnz8pHdnOcAY3obRbSnWWZ8WrjQSGCfqSLOLtSA6Twt 809wCtxWNOyL3eu0LJFmed3tWjmOy7EvqHQIxul+vBuArshODRf0orgs9yFSevgTIgTj cOJcPDFNKuDbSkBrO5nWGmrmHhT/NidNVYPRUA6H+amDmdgI60+FKbYbALMY9wDZcV3F Z5VNdAsyQOJxgmaOeM6q3bYDA5CJwQurtpdFiAkbwlNMy6KluiALTTSC9oNXpH0W0RYq fhG9rTXGvfJOot+CNLof9Ve0EoB5A0gmC4zonJQ02DlWIZw8UVNsL2u5/SViEjuWTQZq 7BgQ== X-Gm-Message-State: AOJu0YyzhENePZHa96ln5KwbNExygj4x2kjlLT6qE2xmruYp4lvq65T3 4rlS8oQcOs1V7Td1lXbvlSu61xy86qqKJw/MOFsMKWKTSws134stiSFZKLJCbIN4r//kdyHDKv+ 4/kYXrCkqzyTumX5RZa9hARBsok28iq6pbNCqjeCanGlJJGdSNTLUP+86iH11ci51JsMM7CY1Tn U= X-Google-Smtp-Source: AGHT+IEv0Jf3vHaRCycMnBj5VeR2nnJQKH9V6EYuLNmnaVoxelKGumyrc8C1NGPQ35rU7hR2Vwwlx8le X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:4e4b:b0:409:6edc:8a07 with SMTP id e11-20020a05600c4e4b00b004096edc8a07mr44734wmq.4.1700821231718; Fri, 24 Nov 2023 02:20:31 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:48 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2460; i=ardb@kernel.org; h=from:subject; bh=wwOsRm3FZFgGJ9li8cLLVrn99cMWNOQlYRvDxthbRTc=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhonORgn5Unf+zP/m2Qsd0H3BM7HP/sTJOMUZ2ke/hy v8FNXEdpSwMYhwMsmKKLAKz/77beXqiVK3zLFmYOaxMIEMYuDgF4CJijAyvPzwL8WplrLm1ef+S b+0cv/9dDda4k3DP0JN9a+Qct7w6hv8xC58GHIkxZJ2Qusf4lYGI3VHWh1GMi6L22ccv97x3iZs dAA== X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-49-ardb@google.com> Subject: [PATCH v5 08/39] arm64: vmemmap: Avoid base2 order of struct page size to dimension region From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022033_532797_80317742 X-CRM114-Status: GOOD ( 16.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The placement and size of the vmemmap region in the kernel virtual address space is currently derived from the base2 order of the size of a struct page. This makes for nicely aligned constants with lots of leading 0xf and trailing 0x0 digits, but given that the actual struct pages are indexed as an ordinary array, this resulting region is severely overdimensioned when the size of a struct page is just over a power of 2. This doesn't matter today, but once we enable 52-bit virtual addressing for 4k pages configurations, the vmemmap region may take up almost half of the upper VA region with the current struct page upper bound at 64 bytes. And once we enable KMSAN or other features that push the size of a struct page over 64 bytes, we will run out of VMALLOC space entirely. So instead, let's derive the region size from the actual size of a struct page, and place the entire region 1 GB from the top of the VA space, where it still doesn't share any lower level translation table entries with the fixmap. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/memory.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 2745bed8ae5b..b49575a92afc 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -30,8 +30,8 @@ * keep a constant PAGE_OFFSET and "fallback" to using the higher end * of the VMEMMAP where 52-bit support is not available in hardware. */ -#define VMEMMAP_SHIFT (PAGE_SHIFT - STRUCT_PAGE_MAX_SHIFT) -#define VMEMMAP_SIZE ((_PAGE_END(VA_BITS_MIN) - PAGE_OFFSET) >> VMEMMAP_SHIFT) +#define VMEMMAP_RANGE (_PAGE_END(VA_BITS_MIN) - PAGE_OFFSET) +#define VMEMMAP_SIZE ((VMEMMAP_RANGE >> PAGE_SHIFT) * sizeof(struct page)) /* * PAGE_OFFSET - the virtual address of the start of the linear map, at the @@ -47,8 +47,8 @@ #define MODULES_END (MODULES_VADDR + MODULES_VSIZE) #define MODULES_VADDR (_PAGE_END(VA_BITS_MIN)) #define MODULES_VSIZE (SZ_2G) -#define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT))) -#define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) +#define VMEMMAP_START (VMEMMAP_END - VMEMMAP_SIZE) +#define VMEMMAP_END (-UL(SZ_1G)) #define PCI_IO_START (VMEMMAP_END + SZ_8M) #define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE) #define FIXADDR_TOP (-UL(SZ_8M)) From patchwork Fri Nov 24 10:18:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 45A8CC61DF4 for ; Fri, 24 Nov 2023 10:21:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Q0niNzGLE6dbY3ah0MYiyZM9l7e/j792YaYFbUuINSg=; b=x4xPuW/CnRo3Q0nkil2WnIbCPp MBDblybCxfW+kX9aBempxdzaADxWJoK0qYfzKLmJGkB+14YLMemEAblxPU9INo/bs6rStfSA2kbzn Ei8rWynNIiufdVg7nu0i5FI2ryNI3R5BaCL2IilMABMCtVvkUyZPSulf6DmAfCYXF6LJ5ZGqZlWqL XRsBb34b4wENdjDxiXDgxleOll5HJvAn1YqkWPb/B0qMCYd65necp4L+lK7kYurZACTczmH+sfxus hCuWGq8fy8SYOJYjyjjOqxsEgX873ZjZs2wZGZcqbMbWqohmbNBUQ4PcsCOw9bNH+65mjijhfkfqR YPGvTbTg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIm-006ohF-0E; Fri, 24 Nov 2023 10:20:48 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIZ-006oaP-1z for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:20:36 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-40b3d4d6417so863545e9.0 for ; Fri, 24 Nov 2023 02:20:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821234; x=1701426034; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=B+BEyFXzG6cg5GYg8KNy+ajGmyiq/MJQTumFJr+LLPA=; b=OkikygQPwa5wUKjOjbVQjUTAuECno6hOIqmEiDFhpip6Q95JlFl+IB7kz6B/MioufQ 6pFsoB2izzI4JweJNLTZiLgQ4Oc/jAmm6o4qm/ELb68D3mYEn95PqmZ7paGuR+o6WNd2 FFOcf6A96qf26amSG7Zk0FasiVwYSSej7ldcRhKXQm6uJS68Vy+zMmxnyBT7GphX7EQF /Wk0jKQeNHE2EDnwzQuKOAeAGG717ug1qmYj5HUvophE/xsqJiFBLZcr0YfhEtJiXeAA lxNkWSzulvht46Or7GKwjX/OtusxR2bdouAOWpRJXbcwnAff5YY5g5O/TA6V1/O0Qg63 bD5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821234; x=1701426034; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=B+BEyFXzG6cg5GYg8KNy+ajGmyiq/MJQTumFJr+LLPA=; b=K9k4xaNMKYk8L9oHP5QYf6DkWzTt/CVAu5BHIlVAyQC9Sp1WSFLUouWhfyxDtWx9ij LCqhOv7yn7AyUR7Skt6hyqrLXBhNqWeh3/rMcLQfER0pTcW2BuFkmzjpOZalgG2+EP4p ZcauoPXq+ew89W2YEsTIkFKKzGFwBQXClszdLCyFZs795HKGalk+UQSWIjltCAR6z3CV v16yi0EpDP9oQFnKWMf+vBA88RI9D4OG/+Q8hOim97O56wIEIiUn8MVkKNAPnV4q3afk RzUasm9E/EegfBWpx5VlB0P764DlxYZA2mS/q+nSPOzY/Cg60OGjSjIQK/iyyHmD6rdc 44tw== X-Gm-Message-State: AOJu0YyxGuekZnjuPAJ3iYAtHbHO3USKYTUwoi0p9YhEgTyE98fm+/3I 39PSkv1HRi9wf/ga6tW3RGxyE5/JmC/mZkyGdxTUvLpj9TI+SuF7yhFK1glvbumJcDh+1dbN/eH 9aRljS+lFZNncA/BiUoYugwCFxs9mKPLeasLVS3dlET1NoEphlIt7EGTKFn+ZjNB82SY3SYGYCG c= X-Google-Smtp-Source: AGHT+IELRH4mS5fEGTbNwvQcnxYTfMufn0YZSVafXWjqZFr6XDwjflHFhVtghpKrZLH6Z3yj/WsF//wx X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:5025:b0:40b:2ab2:b343 with SMTP id n37-20020a05600c502500b0040b2ab2b343mr45618wmr.2.1700821233820; Fri, 24 Nov 2023 02:20:33 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:49 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1794; i=ardb@kernel.org; h=from:subject; bh=83HVzL7EQSX7w4jk++s4dOJdvs3TCvaqo4xiMP29vyE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhoutkZdTePc93/1l7bFLn1E+tYp/C1/XpPNwgOif3Y Yv4hNZbHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiqxcxMvycVP8l8n2emNuM r7pnD/a90b3KMM//voOcjtDNuV5uJS6MDJum/VcrUbub+PfntAlz86xPqMjyh0xyX6Qeyt04Lce 2hRkA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-50-ardb@google.com> Subject: [PATCH v5 09/39] arm64: mm: Reclaim unused vmemmap region for vmalloc use From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022035_654248_5CDEC02E X-CRM114-Status: GOOD ( 14.46 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The vmemmap array is statically sized based on the maximum supported size of the virtual address space, but it is located inside the upper VA region, which is statically sized based on the *minimum* supported size of the VA space. This doesn't matter much when using 64k pages, which is the only configuration that currently supports 52-bit virtual addressing. However, upcoming LPA2 support will change this picture somewhat, as in that case, the vmemmap array will take up more than 25% of the upper VA region when using 4k pages. Given that most of this space is never used when running on a system that does not support 52-bit virtual addressing, let's reclaim the unused vmemmap area in that case. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/pgtable.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 8d30e2787b1f..5b5bedfa320e 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -18,11 +18,15 @@ * VMALLOC range. * * VMALLOC_START: beginning of the kernel vmalloc space - * VMALLOC_END: extends to the available space below vmemmap, PCI I/O space - * and fixed mappings + * VMALLOC_END: extends to the available space below vmemmap */ #define VMALLOC_START (MODULES_END) +#if VA_BITS == VA_BITS_MIN #define VMALLOC_END (VMEMMAP_START - SZ_8M) +#else +#define VMEMMAP_UNUSED_NPAGES ((_PAGE_OFFSET(vabits_actual) - PAGE_OFFSET) >> PAGE_SHIFT) +#define VMALLOC_END (VMEMMAP_START + VMEMMAP_UNUSED_NPAGES * sizeof(struct page) - SZ_8M) +#endif #define vmemmap ((struct page *)VMEMMAP_START - (memstart_addr >> PAGE_SHIFT)) From patchwork Fri Nov 24 10:18:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467444 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A52A9C61D97 for ; Fri, 24 Nov 2023 10:21:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=PuV1abELZfVWcfkvztgVYRwOlQSQ4fp+Fmd/SCB8s9g=; b=pL6vRPyJMycEFe2I/69KEATftc /Z8eVOH4PynbmdCz9PcnD+At62QKMqhqdgduP8597b9SUpPpnb2uPQ/1n50RcmQshvNirt7blf/K6 B4NU9xjPa7CxAWF/3CBMY2Dk1VnILOl9OqyF7eg/+m2YdsN/hqE4cmsKSTSQb3L7vZEtMP1KTk+rF gW7QLcJ0SkG8e115KlKQZ33KIiUmD7iVF904A2Ml1R4UO1nsYK3g2sz9S1iXCOuMndv8SltGkOm/v c3ZB9rGce3qvR0yhwOUL0M/SVqomTA7u7/Q+g8NLoWheXfgWsPfWJjSq76lBjC0uRN+s7oy6O4raH xOpYPXnA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIn-006oig-2y; Fri, 24 Nov 2023 10:20:49 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIb-006obD-2u for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:20:39 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-40b3487fa9eso11104555e9.2 for ; Fri, 24 Nov 2023 02:20:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821236; x=1701426036; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TemnQ6kclmDQ1Y/tK0SncTYLfLpErlQSmwcRZLsuZ5M=; b=KMV3J2PAaXypgRKtLkwa8Sn3zvDm03q5EAHYBYbqL3qgnHqdzWC4357WfaQT/R53eW T5FmFYTd18fVnJNxWFdYc5bjOYo4B0NrxEeCXCXiqUQpXVD6dSfDnNuwXwyxBpXmnWxn 7ULcg/HARSrLI0IvACzmn96srBAzJ54uo7MEqrbue14CgfSml8/UNGGZBQSJ+VPteRpe i6kttOnGWSNDVvJJ6Yf13rhh+oHjmNFULWkTypKaDKG/HMgP7KYGhzCvCAr7Cmq/4AWZ 0gECbuKNFrEJLmH3H0Bf+c2pwEVvh2dkMpYgssFivUPxNBuDUT2wHHCqKh+EPtzDHO4z BvrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821236; x=1701426036; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TemnQ6kclmDQ1Y/tK0SncTYLfLpErlQSmwcRZLsuZ5M=; b=Mq04k4gsEuM5xE0jTuoMe49fEwOwg9X90/qYrTM2pvoPlVX4VCbhkUIKg4SpokWB+D CLahakI0euZ7SEGn+zOlMqHyUhBu/G8BARVTZgibzs7+BifqmH+4FWLnIhTlx21AKZjF 3klIlTx0DQAjddPIwypGOsG3Y/I3QhJnqrcOFCeBvm/lQeUXdofZT9DNuS0i0k2l1/Al qcHPgx5oE0PL0JxNpJUB1oPNHM8JI+r4PBxOqWbe2k2G92IMTMgIh4vaNvKT+jg7ikiM T7oEw5ZHIkfgC5exP4nIWjX9zzaimh/JHTYT+S8G40WVlVuzuslIA/uAYPLGVVCiPDgk A8sQ== X-Gm-Message-State: AOJu0YyxYwJKCyiIJyJKLCu9tBvXfRo630w0rpQg3l9bVCWsAbsVGm1n XeqNJsKEjoZOKzVPJprDyo+OXsdLleb9yUJ4gos1KPjRhPUmEEm+9qG/KYSDFtjizUa1Kr3sHPA iWmLw59a5lFaq1wiXGKCwJej8RtwLfsX3LI3Qp2/DyCKl8DXHA/RRTeBOmdDMCVRorLW12w/WNI 0= X-Google-Smtp-Source: AGHT+IEFjhCc/t1AIjdKRtrJgSsH+nEdZdWoCT8lge3NHU9+bMZ1XDoo8ebq3un6/uDqsXiXW1DYirSk X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:c09:b0:40b:2b38:a210 with SMTP id fm9-20020a05600c0c0900b0040b2b38a210mr43490wmb.8.1700821235994; Fri, 24 Nov 2023 02:20:35 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:50 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2767; i=ardb@kernel.org; h=from:subject; bh=b4D2PcJszYXmkQEIBtlIkaYoXvVpy75o5lzK1n09vY8=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhoqeh0arLabmSm/tLuZv22l/Fr+64dlqu+aHfqyU/J KaWrHLpKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABOZtIGRYW6c/LMssbMX2pZ0 Pch4NVMj83zblPrss/6Zz6sPBHX9WsHwv7T5acfGXzte7yy57Gs5I9FuQZBxQrkEZ835n91Tt51 z4wcA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-51-ardb@google.com> Subject: [PATCH v5 10/39] arm64: kaslr: Adjust randomization range dynamically From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022037_936344_99B6707F X-CRM114-Status: GOOD ( 15.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Currently, we base the KASLR randomization range on a rough estimate of the available space in the upper VA region: the lower 1/4th has the module region and the upper 1/4th has the fixmap, vmemmap and PCI I/O ranges, and so we pick a random location in the remaining space in the middle. Once we enable support for 5-level paging with 4k pages, this no longer works: the vmemmap region, being dimensioned to cover a 52-bit linear region, takes up so much space in the upper VA region (the size of which is based on a 48-bit VA space for compatibility with non-LVA hardware) that the region above the vmalloc region takes up more than a quarter of the available space. So instead of a heuristic, let's derive the randomization range from the actual boundaries of the vmalloc region. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/image-vars.h | 2 ++ arch/arm64/kernel/pi/kaslr_early.c | 11 ++++++----- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 5e4dc72ab1bd..e931ce078a00 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -36,6 +36,8 @@ PROVIDE(__pi___memcpy = __pi_memcpy); PROVIDE(__pi___memmove = __pi_memmove); PROVIDE(__pi___memset = __pi_memset); +PROVIDE(__pi_vabits_actual = vabits_actual); + #ifdef CONFIG_KVM /* diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index 17bff6e399e4..b9e0bb4bc6a9 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -14,6 +14,7 @@ #include #include +#include /* taken from lib/string.c */ static char *__strstr(const char *s1, const char *s2) @@ -87,7 +88,7 @@ static u64 get_kaslr_seed(void *fdt) asmlinkage u64 kaslr_early_init(void *fdt) { - u64 seed; + u64 seed, range; if (is_kaslr_disabled_cmdline(fdt)) return 0; @@ -102,9 +103,9 @@ asmlinkage u64 kaslr_early_init(void *fdt) /* * OK, so we are proceeding with KASLR enabled. Calculate a suitable * kernel image offset from the seed. Let's place the kernel in the - * middle half of the VMALLOC area (VA_BITS_MIN - 2), and stay clear of - * the lower and upper quarters to avoid colliding with other - * allocations. + * 'middle' half of the VMALLOC area, and stay clear of the lower and + * upper quarters to avoid colliding with other allocations. */ - return BIT(VA_BITS_MIN - 3) + (seed & GENMASK(VA_BITS_MIN - 3, 0)); + range = (VMALLOC_END - KIMAGE_VADDR) / 2; + return range / 2 + (((__uint128_t)range * seed) >> 64); } From patchwork Fri Nov 24 10:18:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467446 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1DA51C61DF7 for ; Fri, 24 Nov 2023 10:21:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=fFCMDyHhLMz4hpI3JXfIObJ78LKVsLBoqen3krNecQE=; b=rJwB/g3mig9QgXjtBS+NWseTOK YiauPDE6qD85j4DifR2ED8rTACTrfQthH8f0dEILdXJPs3NJrEus27Isk+JZIWTwFgCHzkVnFN6xF pIGtIofcc/PfwNE7Xfar4D0XsbOt05SP6ncPni3NxpRcYi56lSq1pfmLWIY0ZEMU9RzrkQDxqPRyJ 9CDa1+P5fnOeRG3W+Y7pwfTVS3ngv4eoe9lUqmXNMpVAMk98m0byTQDGLwHC4DUCRJfJeGkBdGW4T Wson8DD8bE21oukmzmSi8ZKiKLwM4fJx3xxgsKABUHOIkaO74+bqCWLODvKvc86T5ksRjjd2a9Tkk A5h6LmGQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIp-006ojq-2R; Fri, 24 Nov 2023 10:20:51 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIe-006ocE-1T for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:20:42 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-40b349b9711so11208085e9.0 for ; Fri, 24 Nov 2023 02:20:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821239; x=1701426039; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=fg5c2ES5qVmnvPkCIq26yaEBq8tFnKPTZ8+FRHt4Ah0=; b=HYGv+NszVwigCMorjEajqVQtjgLG6O0b74yV5eafMjqdMJT09yFkeAeMlvebHTD1Uf bzymanoOdKiKHEfQEnVNLMCGFeBJ7hCXiLELh4BFp45+OIkW178BXbwNbUDdu/z06wb6 NzRsUabfXa6NCBwEn/MYEufA8rvLDg+lL4Ia/LhyWjU85Ha0rjbWK4jEarC5evvV7exR eDh0KUJU1njPGg/XyWoUBLf/oFGgFdlMelusnHpr5+iUe77zw0I9i7ogU5Jexy9OtGTO usNDS9SA1EY478pd3xPxoXJoXgeBFyM7ptczBK0w2HhquVWK/ZH878kWQ9ITZmZL/nC9 qJsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821239; x=1701426039; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fg5c2ES5qVmnvPkCIq26yaEBq8tFnKPTZ8+FRHt4Ah0=; b=ZPR11tJ9vJSqCAdjOwUDO1DHLtLT8S/sDu2UJBTmDPLIAsV71ur7UHfXm050VG9T/9 WXfIXhnxLX5cxrHzc1Z2e4TD2zVlCCDd+THG1beq6/A9TzXTNJscQtSQNyARKp/8TkVo u3Ai3oF3zbgdGLc3+0PwVCG9vcLarwxpdeRHhqK5YZb/7wvF/vfuiIhHgaQ87LLSJMuE LOtDsPq9HNfbu5qG/U7jSzj1HsFOygPX9HKjjOhWYUp1lKsiTFFG7XnHA0dqNNyzDMOF bJqZPKnmqYJbEcFPKhFOUmuRPU97wok6l0VM8UvOpE5vLgmkFqNF0hUnHUYKBUuQDiyP KcqQ== X-Gm-Message-State: AOJu0YwHMyiNA+074zd5zPtTG22GKyPPTwHO6VGqkvHoxsBNou0EEMU6 ersvztSIW+1rDKUZ6SAhQ2m93YHPwDVUXNlFmlhgv3kGqlu91b9AjwrGwHRWLVrQlnP6XsiSMPx hyDoXVO0zGKZxCNuDfaD4VGCgWGT5SKD35Yuu0XTxwnuMkfBORXsnykspnBkjfP04bXCS3tQAtn E= X-Google-Smtp-Source: AGHT+IHMVDBxJVYNCTfHMSaNYH+paB2ujjCiq/sB8Bw00kmc3hmIw2b/pXn+1ie5g/1GAs2Ro0t4lBoj X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:c09:b0:40b:2b38:a210 with SMTP id fm9-20020a05600c0c0900b0040b2b38a210mr43496wmb.8.1700821238356; Fri, 24 Nov 2023 02:20:38 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:51 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6467; i=ardb@kernel.org; h=from:subject; bh=8I8ba1oxqAB2ICxL0NDw8FXdw0Opex7o96s//GRPp84=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWholfUk7ehVVD3uK1QxibJVbvje4PW8cqsFWb/VPP70 OZ7QdUdpSwMYhwMsmKKLAKz/77beXqiVK3zLFmYOaxMIEMYuDgFYCImvowMNyt6D5Zv2LZvQon2 3CwXxjjd2XNcuroV1J26fm9fGzZHguEn44xc9wjePVw7/vQKLVu8alfKw3u1KaGn+jb9XnuOLcu UEQA= X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-52-ardb@google.com> Subject: [PATCH v5 11/39] arm64: kernel: Manage absolute relocations in code built under pi/ From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022040_495368_F5962060 X-CRM114-Status: GOOD ( 23.70 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The mini C runtime runs before relocations are processed, and so it cannot rely on statically initialized pointer variables. Add a check to ensure that such code does not get introduced by accident, by going over the relocations in each object, identifying the ones that operate on data sections that are part of the executable image, and raising an error if any relocations of type R_AARCH64_ABS64 exist. Note that such relocations are permitted in other places (e.g., debug sections) and will never occur in compiler generated code sections when using the small code model, so only check sections that have SHF_ALLOC set and SHF_EXECINSTR cleared. To accommodate cases where statically initialized symbol references are unavoidable, introduce a special case for ELF input data sections that have ".rodata.prel64" in their names, and in these cases, instead of rejecting any encountered ABS64 relocations, convert them into PREL64 relocations, which don't require any runtime fixups. Note that the code in question must still be modified to deal with this, as it needs to convert the 64-bit signed offsets into absolute addresses before use. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/pi/Makefile | 9 +- arch/arm64/kernel/pi/pi.h | 14 +++ arch/arm64/kernel/pi/relacheck.c | 130 ++++++++++++++++++++ 3 files changed, 151 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index c844a0546d7f..bc32a431fe35 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -22,11 +22,16 @@ KCSAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n +hostprogs := relacheck + +quiet_cmd_piobjcopy = $(quiet_cmd_objcopy) + cmd_piobjcopy = $(cmd_objcopy) && $(obj)/relacheck $(@) $(<) + $(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ \ --remove-section=.note.gnu.property \ --prefix-alloc-sections=.init -$(obj)/%.pi.o: $(obj)/%.o FORCE - $(call if_changed,objcopy) +$(obj)/%.pi.o: $(obj)/%.o $(obj)/relacheck FORCE + $(call if_changed,piobjcopy) $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h new file mode 100644 index 000000000000..f455ad385976 --- /dev/null +++ b/arch/arm64/kernel/pi/pi.h @@ -0,0 +1,14 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Author: Ard Biesheuvel + +#define __prel64_initconst __section(".init.rodata.prel64") + +typedef volatile signed long prel64_t; + +static inline void *prel64_to_pointer(const prel64_t *offset) +{ + if (!*offset) + return NULL; + return (void *)offset + *offset; +} diff --git a/arch/arm64/kernel/pi/relacheck.c b/arch/arm64/kernel/pi/relacheck.c new file mode 100644 index 000000000000..b0cd4d0d275b --- /dev/null +++ b/arch/arm64/kernel/pi/relacheck.c @@ -0,0 +1,130 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2023 - Google LLC + * Author: Ard Biesheuvel + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ +#define HOST_ORDER ELFDATA2LSB +#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__ +#define HOST_ORDER ELFDATA2MSB +#endif + +static Elf64_Ehdr *ehdr; +static Elf64_Shdr *shdr; +static const char *strtab; +static bool swap; + +static uint64_t swab_elfxword(uint64_t val) +{ + return swap ? __builtin_bswap64(val) : val; +} + +static uint32_t swab_elfword(uint32_t val) +{ + return swap ? __builtin_bswap32(val) : val; +} + +static uint16_t swab_elfhword(uint16_t val) +{ + return swap ? __builtin_bswap16(val) : val; +} + +int main(int argc, char *argv[]) +{ + struct stat stat; + int fd, ret; + + if (argc < 3) { + fprintf(stderr, "file arguments missing\n"); + exit(EXIT_FAILURE); + } + + fd = open(argv[1], O_RDWR); + if (fd < 0) { + fprintf(stderr, "failed to open %s\n", argv[1]); + exit(EXIT_FAILURE); + } + + ret = fstat(fd, &stat); + if (ret < 0) { + fprintf(stderr, "failed to stat() %s\n", argv[1]); + exit(EXIT_FAILURE); + } + + ehdr = mmap(0, stat.st_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + if (ehdr == MAP_FAILED) { + fprintf(stderr, "failed to mmap() %s\n", argv[1]); + exit(EXIT_FAILURE); + } + + swap = ehdr->e_ident[EI_DATA] != HOST_ORDER; + shdr = (void *)ehdr + swab_elfxword(ehdr->e_shoff); + strtab = (void *)ehdr + + swab_elfxword(shdr[swab_elfhword(ehdr->e_shstrndx)].sh_offset); + + for (int i = 0; i < swab_elfhword(ehdr->e_shnum); i++) { + unsigned long info, flags; + bool prel64 = false; + Elf64_Rela *rela; + int numrela; + + if (swab_elfword(shdr[i].sh_type) != SHT_RELA) + continue; + + /* only consider RELA sections operating on data */ + info = swab_elfword(shdr[i].sh_info); + flags = swab_elfxword(shdr[info].sh_flags); + if ((flags & (SHF_ALLOC | SHF_EXECINSTR)) != SHF_ALLOC) + continue; + + /* + * We generally don't permit ABS64 relocations in the code that + * runs before relocation processing occurs. If statically + * initialized absolute symbol references are unavoidable, they + * may be emitted into a *.rodata.prel64 section and they will + * be converted to place-relative 64-bit references. This + * requires special handling in the referring code. + */ + if (strstr(strtab + swab_elfword(shdr[info].sh_name), + ".rodata.prel64")) { + prel64 = true; + } + + rela = (void *)ehdr + swab_elfxword(shdr[i].sh_offset); + numrela = swab_elfxword(shdr[i].sh_size) / sizeof(*rela); + + for (int j = 0; j < numrela; j++) { + uint64_t info = swab_elfxword(rela[j].r_info); + + if (ELF64_R_TYPE(info) != R_AARCH64_ABS64) + continue; + + if (prel64) { + /* convert ABS64 into PREL64 */ + info ^= R_AARCH64_ABS64 ^ R_AARCH64_PREL64; + rela[j].r_info = swab_elfxword(info); + } else { + fprintf(stderr, + "Unexpected absolute relocations detected in %s\n", + argv[2]); + close(fd); + unlink(argv[1]); + exit(EXIT_FAILURE); + } + } + } + close(fd); + return 0; +} From patchwork Fri Nov 24 10:18:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467563 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9B363C61DF4 for ; Fri, 24 Nov 2023 11:31:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=TgQX1g53M4tLZd8X7EtiH8g/GHkop4FNSdTN5XfOnP8=; b=OG5aCqJb2LeUkMTStmRlwrcQPX 7H/4XbZ6pL1k++7tXEj+Kf3SzhrQjklnNaOFGCwSSDVG2Df2BMTU5EyXSBmM5knnm9xJFHwayBVEy RUd/p7OuaywY2yvjAaV3Cbh16vBLEZRCD7kn8UgR3b82iJT4W0AaQZTvOhnkIXe/2uhxx1mIjmHal 16CDuLDe0SpTbuaNwci2KXc5AZODgZdNoz3kavM6jgQSGG3TEb7vSGf2YS1zGPFrW/b8KigrFVFWT K2minoz+aeci/ZburZl+MNbgb5txcOurFJhYeGJegluKEz3MjEPSJOIEvHmAc/SjRe1DFz4swt5Wl jA2LdOtA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6UPT-006ySB-2S; Fri, 24 Nov 2023 11:31:47 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIg-006od2-2P for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:20:44 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-db3ef4c7094so1902474276.1 for ; Fri, 24 Nov 2023 02:20:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821241; x=1701426041; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4Z62d9o+cY1n/Rvw0hfzV44XBFWrFN972rVNZfEcjMs=; b=eizRHP+8J10+9LhvB+++kLp7mD5PYE62xlVDWX4IPKBJtfCakq1o8Tg/Rwwre58ZCH X1HGj4vImoq/hX+LxPYKl+lc2SEno0GEajkwKOJZxjV6ZFoHrjEL0rZmb10019rhGf1s +Uj/XUfxVPu40PPD5apJGVVfiGgSw/n4j52zaMVYrr2sl35090PlRLE2sKiFkXeaRCAA xRZJVSCdZRQ+CnzNQn/Otua8h+jEc+kdSGdLDHQrMOJs24tqkfvXz+w6EWShsmGqV+b/ RBBCe0TAPdWgG7aHzp0dK1Sp1zDzXb3sgVT/oa4tVA0FVaQZ0VlcUpG9dLv9JAOeIYNC uUyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821241; x=1701426041; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4Z62d9o+cY1n/Rvw0hfzV44XBFWrFN972rVNZfEcjMs=; b=PGABOcxzSL+rGXkJO6n6GpvT/lI9IJOcPlyRel95u1hVdTZ8eVHkIMqiaeNyMLUIPb llZ+/YXGzlIjrctqY95H45aRJ442qlkarYbvMtT3V7xYWi1KiJ3acp4s/k/aNGxPJDHX jZ2bJVg/wJUk9DFHsrcRJEOyjuL7G6i5BL9WEbDIcD3+u9mv+s89BB8pytPSnfUUX9sm QQAxFRsdENAkfjxmTaNyZ7Aivjd7EiEkP7mTIytpxcJUF4NYEp75lEDXkSr99Smt9NQg Jphb3+H2Ek9xa11D1sltupTJgMiLqnSVsS3vnRh9lauwmpOyMcyFySGyQUCnE7ZylgZF 9BKQ== X-Gm-Message-State: AOJu0YwMRDfsccO3mb21Zo6Kbc8UFvs5cZAoOO6YBGfLvbf11mWdZHtg bHBGQsRK7mHUyVBPJU7wqRrdPtx6u6c/sMM4tuRdsPHzSj6/9ApzmzpBvzTvpNQWMVIcjrGJuH5 shgd9ElSmViq56nZ2gtSVd1sePd18AByir9VUyb0ijT1GSP3IjLZBXaDca7QYSFEDv0J0fPuE2r 0= X-Google-Smtp-Source: AGHT+IGJUGb0nqwq2a63gUogmCplOh6FPttRC5y6N+Q8Qjch2JHlRrMYOfM97ERQhEozqxkC4MFiJ2g4 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:ab8b:0:b0:db4:5dc7:1aea with SMTP id v11-20020a25ab8b000000b00db45dc71aeamr55028ybi.6.1700821241123; Fri, 24 Nov 2023 02:20:41 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:52 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3215; i=ardb@kernel.org; h=from:subject; bh=fPQie6cXvO/xVNq+vsDwiE5UF5vcdZhEaJpXnkvDOvk=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhou/Rju3/PMvtpr2fz+11eWnrzRNl396XxJ/l9Vbfs Z43YWVyRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZhI3kdGhh2TP5b++DP9fl2O vdWxD7X+G489/rXnWEh8Q5HykhM1O2UYGT7sYl9vHHbm5sf+by1RbE/y3gUuM162dNalU3+Pxl5 atosFAA== X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-53-ardb@google.com> Subject: [PATCH v5 12/39] arm64: kernel: Don't rely on objcopy to make code under pi/ __init From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022042_789590_6CDBC919 X-CRM114-Status: GOOD ( 12.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We will add some code under pi/ that contains global variables that should not end up in __initdata, as they will not be writable via the initial ID map. So only rely on objcopy for making the libfdt code __init, and use explicit annotations for the rest. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/pi/Makefile | 6 ++++-- arch/arm64/kernel/pi/kaslr_early.c | 16 +++++++++------- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index bc32a431fe35..2bbe866417d4 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -28,11 +28,13 @@ quiet_cmd_piobjcopy = $(quiet_cmd_objcopy) cmd_piobjcopy = $(cmd_objcopy) && $(obj)/relacheck $(@) $(<) $(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ \ - --remove-section=.note.gnu.property \ - --prefix-alloc-sections=.init + --remove-section=.note.gnu.property $(obj)/%.pi.o: $(obj)/%.o $(obj)/relacheck FORCE $(call if_changed,piobjcopy) +# ensure that all the lib- code ends up as __init code and data +$(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init + $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index b9e0bb4bc6a9..167081b30a15 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -17,7 +17,7 @@ #include /* taken from lib/string.c */ -static char *__strstr(const char *s1, const char *s2) +static char *__init __strstr(const char *s1, const char *s2) { size_t l1, l2; @@ -33,7 +33,7 @@ static char *__strstr(const char *s1, const char *s2) } return NULL; } -static bool cmdline_contains_nokaslr(const u8 *cmdline) +static bool __init cmdline_contains_nokaslr(const u8 *cmdline) { const u8 *str; @@ -41,7 +41,7 @@ static bool cmdline_contains_nokaslr(const u8 *cmdline) return str == cmdline || (str > cmdline && *(str - 1) == ' '); } -static bool is_kaslr_disabled_cmdline(void *fdt) +static bool __init is_kaslr_disabled_cmdline(void *fdt) { if (!IS_ENABLED(CONFIG_CMDLINE_FORCE)) { int node; @@ -67,17 +67,19 @@ static bool is_kaslr_disabled_cmdline(void *fdt) return cmdline_contains_nokaslr(CONFIG_CMDLINE); } -static u64 get_kaslr_seed(void *fdt) +static u64 __init get_kaslr_seed(void *fdt) { + static char const chosen_str[] __initconst = "chosen"; + static char const seed_str[] __initconst = "kaslr-seed"; int node, len; fdt64_t *prop; u64 ret; - node = fdt_path_offset(fdt, "/chosen"); + node = fdt_path_offset(fdt, chosen_str); if (node < 0) return 0; - prop = fdt_getprop_w(fdt, node, "kaslr-seed", &len); + prop = fdt_getprop_w(fdt, node, seed_str, &len); if (!prop || len != sizeof(u64)) return 0; @@ -86,7 +88,7 @@ static u64 get_kaslr_seed(void *fdt) return ret; } -asmlinkage u64 kaslr_early_init(void *fdt) +asmlinkage u64 __init kaslr_early_init(void *fdt) { u64 seed, range; From patchwork Fri Nov 24 10:18:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467566 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 297BFC636CB for ; Fri, 24 Nov 2023 11:31:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=cPbEMn09YUAuXkcwE+4o1RIYMewB4wplVqILb10iYLs=; b=I5cA0v5jmA2fipm3V4isyvTlT8 2Qr/fhMBdv4c74xTKAgT8qSFRH/G8zOyXP0zsm+mm9oIXTKWOHvsSrqWmmVufzqXuzi+lzHIM5F+C RcugAS+eHe4lpiiDmM/ZxqCyu5LS3BuholEd7c9HY2yHq1cKd4cGaH5vBGUuXotXFJ8t8F4O6o3Zl zDkIjLAciYeLRpfHaKVM5OHQ4tuEask6NhAkct8sDSAk62/YMnFR72/PsCbOv6n1BeEbqMZfJZ0Dn fBaLtwIz7F+ekkIavedTrUcDEvv6us1tVXmmbQfV/C9MndMmV4TVZEYc4ryNvTrp5VLWo7R1OAdfO XImYOV8Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6UPU-006ySW-0v; Fri, 24 Nov 2023 11:31:48 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIj-006oe4-0F for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:20:48 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-332e592f28aso635680f8f.1 for ; Fri, 24 Nov 2023 02:20:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821243; x=1701426043; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UXY7kxlLq0+SKvm2b1Qbe8FEzV/bS5L72AgeqhFmu2M=; b=Izp+DylXZZHsrO02GwrSBQ2iJjboPAFSWB4TG/P0+fBb7n/l9wgkrMSpNrS3n5N+F3 /gICKh9+v7oNrE+/5Fs2AxL5vLmDR75n4hjFHdmoWBvr+Iv+RiB2JWlv5R5BKXqgBPha OP6MjYgc+yeb4VrFhc5YIRqmtH2IBIaw84MTTxNJYnbAzymN8yuhANKG03g5hnWKhhwu 1E2es5knGKzAL1+9v7ildhKAxZIJgNkKOxPVAiqBAtFL6B9RlFACcX8oCADiqhHbQ8Ti 45xGG931nXGT6mCKfYVdG7jpWOCnLrDzsIhf634cXAd3FY9o8glWJ5RO7ueHEmwlnzq3 mzrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821243; x=1701426043; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UXY7kxlLq0+SKvm2b1Qbe8FEzV/bS5L72AgeqhFmu2M=; b=lUhzgeFWTbg3i1mPFTr9llrZyJPVe5Itfx57liLynCnWL9Nyu1O4u9zNX9XsUfTvhp omYS3uwelCQ057rnci7vsVwh0GIH6hcR+aN/k2EsI+PidVaqN3kvBRCZdgQeBV+7k8J8 BUGz4sImmFfFemoPRYi11wS4u4TB2g46tI/DCtJM6JEVIK2QbbAcBZGp7fwUwmVDfH6K 4X3POloIP1uKtPIlwZRKTMPeMtK3DRIeXfuMIOLzJGHOxhgBfCnCc3pQWACDKZePmSXu +Uf9A/k9k1s1rEiluWieQcJ6TVhtjoXced5RLJvcZNakIBCjqGp0ZPP1FFikR0ckRuEF T01g== X-Gm-Message-State: AOJu0YwFFz3R4DNvR6unvY5Ntq4+CZYrL/jTcrEIpNvR3n5/ter2lXut puRFf+pUVho8LLZzDf5xk0DVk1N9p0YBGHpcff0iwgx8NM2gIWxmIGrDB15YgUg4xARfFjRF8GZ kzI0dR3mZTpLOls8pw0D0+cWbP0lDHhK3B0h4rFNDvlzFiA5BOKL8anLb4HpYe2JoKyJHtvvz+V c= X-Google-Smtp-Source: AGHT+IF4O1gaKMADsluFDIPmUAxVjvZ3slEopdybbb3Txdk8ZC3IhbmFVpGm2ywb+6vxqzSHIzImTlsP X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:4f03:0:b0:332:e5fb:f9e0 with SMTP id c3-20020a5d4f03000000b00332e5fbf9e0mr31052wru.7.1700821243322; Fri, 24 Nov 2023 02:20:43 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:53 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10138; i=ardb@kernel.org; h=from:subject; bh=R8mw+UDux9ZKcsy1xD52mmPjXwN4aNw65KRh2eHMcus=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhor9Z/nTuBKavAbO5Lry0OnVdJPTCMfZJcv7JT56uP BzgeOF6RykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZiI0htGhgfKQX7fTrL5imv1 KzdezXYS7+/xmrmr0+iYhGtgtpeJCsNfYc/+opgX09TKPQ+ff+3zv2ZN6ULG0Ns53ryiesfXfp7 MAQA= X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-54-ardb@google.com> Subject: [PATCH v5 13/39] arm64: head: move relocation handling to C code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022045_129261_B03378A0 X-CRM114-Status: GOOD ( 27.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that we have a mini C runtime before the kernel mapping is up, we can move the non-trivial relocation processing code out of head.S and reimplement it in C. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/Makefile | 3 +- arch/arm64/kernel/head.S | 104 ++------------------ arch/arm64/kernel/pi/Makefile | 5 +- arch/arm64/kernel/pi/relocate.c | 62 ++++++++++++ arch/arm64/kernel/vmlinux.lds.S | 12 ++- 5 files changed, 82 insertions(+), 104 deletions(-) diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index d95b3d6b471a..a8487749cabd 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -57,7 +57,8 @@ obj-$(CONFIG_ACPI) += acpi.o obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL) += acpi_parking_protocol.o obj-$(CONFIG_PARAVIRT) += paravirt.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o pi/ +obj-$(CONFIG_RELOCATABLE) += pi/ +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_ELF_CORE) += elfcore.o obj-$(CONFIG_KEXEC_CORE) += machine_kexec.o relocate_kernel.o \ diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index cab7f91949d8..a8fa64fc30d7 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -81,7 +81,7 @@ * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob - * x23 primary_entry() .. start_kernel() physical misalignment/KASLR offset + * x23 __primary_switch() physical misalignment/KASLR offset * x24 __primary_switch() linear map KASLR seed * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register @@ -389,7 +389,7 @@ SYM_FUNC_START_LOCAL(create_idmap) /* Remap the kernel page tables r/w in the ID map */ adrp x1, _text adrp x2, init_pg_dir - adrp x3, init_pg_end + adrp x3, _end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 mov_q x5, SWAPPER_RW_MMUFLAGS mov x6, #SWAPPER_BLOCK_SHIFT @@ -779,97 +779,6 @@ SYM_FUNC_START_LOCAL(__no_granule_support) b 1b SYM_FUNC_END(__no_granule_support) -#ifdef CONFIG_RELOCATABLE -SYM_FUNC_START_LOCAL(__relocate_kernel) - /* - * Iterate over each entry in the relocation table, and apply the - * relocations in place. - */ - adr_l x9, __rela_start - adr_l x10, __rela_end - mov_q x11, KIMAGE_VADDR // default virtual offset - add x11, x11, x23 // actual virtual offset - -0: cmp x9, x10 - b.hs 1f - ldp x12, x13, [x9], #24 - ldr x14, [x9, #-8] - cmp w13, #R_AARCH64_RELATIVE - b.ne 0b - add x14, x14, x23 // relocate - str x14, [x12, x23] - b 0b - -1: -#ifdef CONFIG_RELR - /* - * Apply RELR relocations. - * - * RELR is a compressed format for storing relative relocations. The - * encoded sequence of entries looks like: - * [ AAAAAAAA BBBBBBB1 BBBBBBB1 ... AAAAAAAA BBBBBB1 ... ] - * - * i.e. start with an address, followed by any number of bitmaps. The - * address entry encodes 1 relocation. The subsequent bitmap entries - * encode up to 63 relocations each, at subsequent offsets following - * the last address entry. - * - * The bitmap entries must have 1 in the least significant bit. The - * assumption here is that an address cannot have 1 in lsb. Odd - * addresses are not supported. Any odd addresses are stored in the RELA - * section, which is handled above. - * - * Excluding the least significant bit in the bitmap, each non-zero - * bit in the bitmap represents a relocation to be applied to - * a corresponding machine word that follows the base address - * word. The second least significant bit represents the machine - * word immediately following the initial address, and each bit - * that follows represents the next word, in linear order. As such, - * a single bitmap can encode up to 63 relocations in a 64-bit object. - * - * In this implementation we store the address of the next RELR table - * entry in x9, the address being relocated by the current address or - * bitmap entry in x13 and the address being relocated by the current - * bit in x14. - */ - adr_l x9, __relr_start - adr_l x10, __relr_end - -2: cmp x9, x10 - b.hs 7f - ldr x11, [x9], #8 - tbnz x11, #0, 3f // branch to handle bitmaps - add x13, x11, x23 - ldr x12, [x13] // relocate address entry - add x12, x12, x23 - str x12, [x13], #8 // adjust to start of bitmap - b 2b - -3: mov x14, x13 -4: lsr x11, x11, #1 - cbz x11, 6f - tbz x11, #0, 5f // skip bit if not set - ldr x12, [x14] // relocate bit - add x12, x12, x23 - str x12, [x14] - -5: add x14, x14, #8 // move to next bit's address - b 4b - -6: /* - * Move to the next bitmap's address. 8 is the word size, and 63 is the - * number of significant bits in a bitmap entry. - */ - add x13, x13, #(8 * 63) - b 2b - -7: -#endif - ret - -SYM_FUNC_END(__relocate_kernel) -#endif - SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir @@ -877,11 +786,11 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RELOCATABLE adrp x23, KERNEL_START and x23, x23, MIN_KIMG_ALIGN - 1 -#ifdef CONFIG_RANDOMIZE_BASE - mov x0, x22 - adrp x1, init_pg_end + adrp x1, early_init_stack mov sp, x1 mov x29, xzr +#ifdef CONFIG_RANDOMIZE_BASE + mov x0, x22 bl __pi_kaslr_early_init and x24, x0, #SZ_2M - 1 // capture memstart offset seed bic x0, x0, #SZ_2M - 1 @@ -894,7 +803,8 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, init_pg_dir load_ttbr1 x1, x1, x2 #ifdef CONFIG_RELOCATABLE - bl __relocate_kernel + mov x0, x23 + bl __pi_relocate_kernel #endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 2bbe866417d4..d084c1dcf416 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,5 +38,6 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o -extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) +obj-y := relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/pi/relocate.c b/arch/arm64/kernel/pi/relocate.c new file mode 100644 index 000000000000..1853408ea76b --- /dev/null +++ b/arch/arm64/kernel/pi/relocate.c @@ -0,0 +1,62 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Authors: Ard Biesheuvel +// Peter Collingbourne + +#include +#include +#include + +extern const Elf64_Rela rela_start[], rela_end[]; +extern const u64 relr_start[], relr_end[]; + +void __init relocate_kernel(u64 offset) +{ + u64 *place = NULL; + + for (const Elf64_Rela *rela = rela_start; rela < rela_end; rela++) { + if (ELF64_R_TYPE(rela->r_info) != R_AARCH64_RELATIVE) + continue; + *(u64 *)(rela->r_offset + offset) = rela->r_addend + offset; + } + + if (!IS_ENABLED(CONFIG_RELR) || !offset) + return; + + /* + * Apply RELR relocations. + * + * RELR is a compressed format for storing relative relocations. The + * encoded sequence of entries looks like: + * [ AAAAAAAA BBBBBBB1 BBBBBBB1 ... AAAAAAAA BBBBBB1 ... ] + * + * i.e. start with an address, followed by any number of bitmaps. The + * address entry encodes 1 relocation. The subsequent bitmap entries + * encode up to 63 relocations each, at subsequent offsets following + * the last address entry. + * + * The bitmap entries must have 1 in the least significant bit. The + * assumption here is that an address cannot have 1 in lsb. Odd + * addresses are not supported. Any odd addresses are stored in the + * RELA section, which is handled above. + * + * With the exception of the least significant bit, each bit in the + * bitmap corresponds with a machine word that follows the base address + * word, and the bit value indicates whether or not a relocation needs + * to be applied to it. The second least significant bit represents the + * machine word immediately following the initial address, and each bit + * that follows represents the next word, in linear order. As such, a + * single bitmap can encode up to 63 relocations in a 64-bit object. + */ + for (const u64 *relr = relr_start; relr < relr_end; relr++) { + if ((*relr & 1) == 0) { + place = (u64 *)(*relr + offset); + *place++ += offset; + } else { + for (u64 *p = place, r = *relr >> 1; r; p++, r >>= 1) + if (r & 1) + *p += offset; + place += 63; + } + } +} diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 3cd7e76cc562..8dd5dda66f7c 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -270,15 +270,15 @@ SECTIONS HYPERVISOR_RELOC_SECTION .rela.dyn : ALIGN(8) { - __rela_start = .; + __pi_rela_start = .; *(.rela .rela*) - __rela_end = .; + __pi_rela_end = .; } .relr.dyn : ALIGN(8) { - __relr_start = .; + __pi_relr_start = .; *(.relr.dyn) - __relr_end = .; + __pi_relr_end = .; } . = ALIGN(SEGMENT_ALIGN); @@ -317,6 +317,10 @@ SECTIONS init_pg_dir = .; . += INIT_DIR_SIZE; init_pg_end = .; +#ifdef CONFIG_RELOCATABLE + . += SZ_4K; /* stack for the early relocation code */ + early_init_stack = .; +#endif . = ALIGN(SEGMENT_ALIGN); __pecoff_data_size = ABSOLUTE(. - __initdata_begin); From patchwork Fri Nov 24 10:18:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467564 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F40ABC61DF7 for ; Fri, 24 Nov 2023 11:31:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=heR6Q2m2q2c/fpyM4ByDedsD6+ALcokXC0c2w7PcLPw=; b=r6OSQos9a1UechO60WSwM2bXmi +mx7Mf/rUW9nvLq0FLslWUp0EnChzZT0+X5D2yMMuOG1uO33hTs7vLTimpbwLpsw+mfhcn3fKGGXN +Ew7nmvHOrNjGdfVAR21tDaXvfYMVf7t0z86v+f9uyEW/vbjOImarBufxgekxkTyKKeygwuEsnO79 6EU4YfuV+lyDuQe+bMqhY6uC9KgP/0oZzH0V5R0y+P+2Y+yIDSeG46qLqOydG09WOQy10RPjCLfiX rWu+cIOkEVhSiMc5Z+EYjAbDpfkiADrBqgERNg5vTkfyw/cCA/r0W5mPvwGU57N1hiJKKmEBPOq5L 2DNxNv4A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6UPU-006ySq-2d; Fri, 24 Nov 2023 11:31:48 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIl-006ofX-2M for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:20:50 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-332cb6be78fso931638f8f.0 for ; Fri, 24 Nov 2023 02:20:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821245; x=1701426045; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4UHjHe6FpBX2R/nrdj2QZAPajpngeY6jPATfzVcbFAU=; b=ItKde8mt9hCRPHGqdAlP+io01jgnnRaET9sFFWNI2HxrOAj1ZTpjwl1oHvenCSOnlG reUNyb/ZlajlxaTEndPBMXr5U+VeHKhWSupZUx/WeYhn4IvG8cqDc9qwOyDkKkffe9G0 5EvFeJ0zxocdez+c/IxjyeyR0ID98O1/cHGZE3i3tGD/yceZhKRfaoQS3RWWmjD06M+K XtXKlZ8R1Pl+nyM8PJU/9EICsNO0ftNVNdravidJzgvoTAO4iDY21d2bIcZtszQwf9jq arI/kw1Fm0Jkosm0Rw3ZtzA4rc/7kqxydNVdB3BuxNMI3Q/2pi8nO8iQRGIXtJKT6Lf7 vogw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821245; x=1701426045; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4UHjHe6FpBX2R/nrdj2QZAPajpngeY6jPATfzVcbFAU=; b=A+jLd5mVLjMLNcDvrhnL3EnUQLNU0cDGueFaNpNO6N2fTdbKvpeii6UTsDzf0ds/eM XisWWAhiZJsxhToP9eRWK5EamM3hIw2w4z99Eh+f1s/m8K5sP3bSwUQKGskFTbThFQwM CU3HTbHmmNaUesPT/JFOylrBeA7Vm3V0oPT4WJuAqMH77Sixhq0XFsJzqLexpPkxo4Rs /vA5spsUGl5cFGVj/eRGLsIQMy1LFkeZJhRF/1bkY1bMAA8fIoEVzl4jaDYLLh+LEmx3 KWztVQ36YuSBp8AGYRHQ9kouq1VWDXQUwgyS++me1ulvqiYejKr5ilwMZy1eBsAKIqJp fgUg== X-Gm-Message-State: AOJu0YwsxoBQnLReOhNwh8iFpfsRigz6naWwviZAl+8R2Qxi7aTIt9I3 7ChQrYrYEjMjgozjhV9HByKmcAYiOrJ9G9Cax1ObzDXSAmx6GPK1YXtIH/a4V7ayelIADDaU2Pq ba6qLfF33VQ8+MS98jJa04d1zTBS/4ovc5YZAPoHdQVpGv+9oxzLREnB8074Fft0+ApmIbIRTut Q= X-Google-Smtp-Source: AGHT+IFfBfp/ISOMg9SMOE3RVxbZ70tLeitDCjr5vAVS8ubiPa7/5hkq/aTcKYUGDqNp+7JoDtGU7Bi9 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:ce10:0:b0:332:e67f:ce87 with SMTP id p16-20020adfce10000000b00332e67fce87mr36121wrn.11.1700821245546; Fri, 24 Nov 2023 02:20:45 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:54 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1689; i=ardb@kernel.org; h=from:subject; bh=3dYQ52MFTwqSSF83zbcTUGZxpjv7uRO4dzR3QdL/tgA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYkJVrWHgmexEjR4Fg5k8+cUmOb2eMhpRpjPKvtS+P n7ozLKOUhYGMQ4GWTFFFoHZf9/tPD1RqtZ5lizMHFYmkCEMXJwCMJGi2YwMnWfbjwsJnAud/1og 4tG7q0sqt2mGcv2Xb8+q2LI//fb73YwM99fw9ludfv5E4uPZmfUT3hwv+yxYFX3iyNXDq5dzOYr eYQMA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-55-ardb@google.com> Subject: [PATCH v5 14/39] arm64: idreg-override: Omit non-NULL checks for override pointer From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022047_790176_3792A22F X-CRM114-Status: GOOD ( 12.72 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that override pointers are always set, we can drop the various non-NULL checks that we have in the code. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index 3addc09f8746..536bc33859bc 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -216,9 +216,6 @@ static void __init match_options(const char *cmdline) for (i = 0; i < ARRAY_SIZE(regs); i++) { int f; - if (!regs[i]->override) - continue; - for (f = 0; strlen(regs[i]->fields[f].name); f++) { u64 shift = regs[i]->fields[f].shift; u64 width = regs[i]->fields[f].width ?: 4; @@ -319,10 +316,8 @@ asmlinkage void __init init_feature_override(u64 boot_status) int i; for (i = 0; i < ARRAY_SIZE(regs); i++) { - if (regs[i]->override) { - regs[i]->override->val = 0; - regs[i]->override->mask = 0; - } + regs[i]->override->val = 0; + regs[i]->override->mask = 0; } __boot_status = boot_status; @@ -330,9 +325,8 @@ asmlinkage void __init init_feature_override(u64 boot_status) parse_cmdline(); for (i = 0; i < ARRAY_SIZE(regs); i++) { - if (regs[i]->override) - dcache_clean_inval_poc((unsigned long)regs[i]->override, - (unsigned long)regs[i]->override + - sizeof(*regs[i]->override)); + dcache_clean_inval_poc((unsigned long)regs[i]->override, + (unsigned long)regs[i]->override + + sizeof(*regs[i]->override)); } } From patchwork Fri Nov 24 10:18:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467567 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 80CB8C636D0 for ; Fri, 24 Nov 2023 11:31:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=EkG4bG5x2lt/sO98bPuD/nxwedXeawDf2Bj/6j/9rug=; b=28HEyYzfnafTn666Giwj0GKuMv guLs5pynx08JfA6rSktNIQ4PLxT3jO3JqtUE/vINXGqkMSvfyYVa5vxBfB6/8DjW48dz5icxhOpXq 25350rUu72lgST4T/x8U4bMjyoJQwse5xgODcT6xwbLZJOeYKQXuveyWNFh8WBynMK3lQLBxDLZJr VgBM1+R0KQCYBB+KM9fXohKETnDraGbZQVO8VDCX6tjyGSwx2g4PVn5BBbaccwZAViCVxxWzjLtPL gqazGmEHbPTkLx169nTkmo/YgGzManUw9GtF6ME+1IWwsbDFeGTRs0rDgbRTL34kW87UU/CB0bUan OK8JS8+Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6UPV-006yTW-2n; Fri, 24 Nov 2023 11:31:49 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIn-006ohd-2I for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:20:57 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-40b357e2a01so8803415e9.1 for ; Fri, 24 Nov 2023 02:20:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821248; x=1701426048; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jyJ9aXJPr1TS0bTOf9aHRSmaKnZpYKNHVy9OuKiC+Do=; b=HnjQ5C9vXNXbctVO4zZfx7RQ5WRAZlv7XRpKQcjVXr7DmzLwPA1EznvWcwYT/NjHTr HdArOAwNvEBafMpX74hIj3eEI/cCA2hLHVnNQtOXtI5RgoRyqDK+fXjrjDFz0sOHEHLz EKqnYvcZ8uqzrksFeKzRu5mpUOpC0Nq5iw+Bq2q7YSLoIHC4sGgd542l/8AbHcJyuC5W hTclGeKweEj9SyvOD5dl7lWHTSWU6p69d+SDBtdUhrLwMwhG+5jJ8HtJoqaHskJIH9jj S5oR/kBmlF/8myvo/zxhmjrk9fZgBbWSFHH1VYgO7g67tjGoSi+fii5uL1fQA5f7Szc8 KPpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821248; x=1701426048; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jyJ9aXJPr1TS0bTOf9aHRSmaKnZpYKNHVy9OuKiC+Do=; b=KfaXb3k1L2ZMF4xvlSc294sSme4yOOTsF5RevqVI1WUsgUcapekjour7sGn8PrrHKC G7WliPh1SRzaWpRWuIaw9jhGD1NYBdaVqrSJiMx+HIA44PGsBgl6tcAzMkQPRIAyU3Gv /5rqJedznKiH0hsUcQ55M2pJjePHccCzn9/skHoqHpdcsSbvE11hsO8X1pLbTtY7idBZ yv9/RzNkIoHvj9hbvjtwFpDBbR6S0r7F/pfeT7IXm+np54iB5i7tyqRDk+Ak9HrQ8wsK 7Pj7vp+2MSIH9ZoiXseZftdRDvEacbe0SFMH9Fxl8fkXEa4RGz8DOpASplb9VHV9ksHp avAg== X-Gm-Message-State: AOJu0Yyz7Plnmm8wHahor2NprOfb9eoiTkOA/QCnwVkk1d3AC4kyKk34 Po7RqfNNHjfHgAgGvVhrM6N+HnoHwLZk5/qiTa0Xpt9BKUl68yiTf6AguYObqrYvHJG78832YJz qvbFi50Wu6YjFHeddjNfBr98J9CPiBRZeV7ix/+PVZLOcBjbaEWKbUftk30rNDHF72edzz3gYRi s= X-Google-Smtp-Source: AGHT+IFOUHM8bDnIZMTKwXnUdhsi3OBeRJrwytyRzGngvnUXMh19tUxhXaJjhsJVDT4LBFuWULCJqYr/ X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:12d2:b0:32d:aa14:875f with SMTP id l18-20020a05600012d200b0032daa14875fmr39013wrx.7.1700821247689; Fri, 24 Nov 2023 02:20:47 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:55 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=7408; i=ardb@kernel.org; h=from:subject; bh=OcGSnfOSPAAXJV7v/uYFRQlCxbOdTLOskF+X8oTTUBM=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYtI7ocuP/n2dWPP4vry97Xv31Rsfaxzqz1u3L9DHQ PZinP/mjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCR9a0M/5QvRnp8OrbcKS7l 0F/hmYvin9xSDWDiXzpFd+mmfoXU2j2MDF+vMB8x0Z921P/CVtV8+XUWOSuyL15g7/6xysndLeJ rASsA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-56-ardb@google.com> Subject: [PATCH v5 15/39] arm64: idreg-override: Prepare for place relative reloc patching From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022049_785909_B59ED08B X-CRM114-Status: GOOD ( 20.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The ID reg override handling code uses a rather elaborate data structure that relies on statically initialized absolute address values in pointer fields. This means that this code cannot run until relocation fixups have been applied, and this is unfortunate, because it means we cannot discover overrides for KASLR or LVA/LPA without creating the kernel mapping and performing the relocations first. This can be solved by switching to place-relative relocations, which can be applied by the linker at build time. This means some additional arithmetic is required when dereferencing these pointers, as we can no longer dereference the pointer members directly. So let's implement this for idreg-override.c in a preliminary way, i.e., convert all the references in code to use a special accessor that produces the correct absolute value at runtime. To preserve the strong type checking for the static initializers, use union types for representing the hybrid quantities. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 98 +++++++++++++------- 1 file changed, 65 insertions(+), 33 deletions(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index 536bc33859bc..4e32a44560bf 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -21,14 +21,32 @@ static u64 __boot_status __initdata; +// temporary __prel64 related definitions +// to be removed when this code is moved under pi/ + +#define __prel64_initconst __initconst + +typedef void *prel64_t; + +static void *prel64_to_pointer(const prel64_t *p) +{ + return *p; +} + struct ftr_set_desc { char name[FTR_DESC_NAME_LEN]; - struct arm64_ftr_override *override; + union { + struct arm64_ftr_override *override; + prel64_t override_prel; + }; struct { char name[FTR_DESC_FIELD_LEN]; u8 shift; u8 width; - bool (*filter)(u64 val); + union { + bool (*filter)(u64 val); + prel64_t filter_prel; + }; } fields[]; }; @@ -46,7 +64,7 @@ static bool __init mmfr1_vh_filter(u64 val) val == 0); } -static const struct ftr_set_desc mmfr1 __initconst = { +static const struct ftr_set_desc mmfr1 __prel64_initconst = { .name = "id_aa64mmfr1", .override = &id_aa64mmfr1_override, .fields = { @@ -70,7 +88,7 @@ static bool __init pfr0_sve_filter(u64 val) return true; } -static const struct ftr_set_desc pfr0 __initconst = { +static const struct ftr_set_desc pfr0 __prel64_initconst = { .name = "id_aa64pfr0", .override = &id_aa64pfr0_override, .fields = { @@ -94,7 +112,7 @@ static bool __init pfr1_sme_filter(u64 val) return true; } -static const struct ftr_set_desc pfr1 __initconst = { +static const struct ftr_set_desc pfr1 __prel64_initconst = { .name = "id_aa64pfr1", .override = &id_aa64pfr1_override, .fields = { @@ -105,7 +123,7 @@ static const struct ftr_set_desc pfr1 __initconst = { }, }; -static const struct ftr_set_desc isar1 __initconst = { +static const struct ftr_set_desc isar1 __prel64_initconst = { .name = "id_aa64isar1", .override = &id_aa64isar1_override, .fields = { @@ -117,7 +135,7 @@ static const struct ftr_set_desc isar1 __initconst = { }, }; -static const struct ftr_set_desc isar2 __initconst = { +static const struct ftr_set_desc isar2 __prel64_initconst = { .name = "id_aa64isar2", .override = &id_aa64isar2_override, .fields = { @@ -128,7 +146,7 @@ static const struct ftr_set_desc isar2 __initconst = { }, }; -static const struct ftr_set_desc smfr0 __initconst = { +static const struct ftr_set_desc smfr0 __prel64_initconst = { .name = "id_aa64smfr0", .override = &id_aa64smfr0_override, .fields = { @@ -149,7 +167,7 @@ static bool __init hvhe_filter(u64 val) ID_AA64MMFR1_EL1_VH_SHIFT)); } -static const struct ftr_set_desc sw_features __initconst = { +static const struct ftr_set_desc sw_features __prel64_initconst = { .name = "arm64_sw", .override = &arm64_sw_feature_override, .fields = { @@ -159,14 +177,17 @@ static const struct ftr_set_desc sw_features __initconst = { }, }; -static const struct ftr_set_desc * const regs[] __initconst = { - &mmfr1, - &pfr0, - &pfr1, - &isar1, - &isar2, - &smfr0, - &sw_features, +static const union { + const struct ftr_set_desc *reg; + prel64_t reg_prel; +} regs[] __prel64_initconst = { + { .reg = &mmfr1 }, + { .reg = &pfr0 }, + { .reg = &pfr1 }, + { .reg = &isar1 }, + { .reg = &isar2 }, + { .reg = &smfr0 }, + { .reg = &sw_features }, }; static const struct { @@ -214,15 +235,20 @@ static void __init match_options(const char *cmdline) int i; for (i = 0; i < ARRAY_SIZE(regs); i++) { + const struct ftr_set_desc *reg = prel64_to_pointer(®s[i].reg_prel); + struct arm64_ftr_override *override; int f; - for (f = 0; strlen(regs[i]->fields[f].name); f++) { - u64 shift = regs[i]->fields[f].shift; - u64 width = regs[i]->fields[f].width ?: 4; + override = prel64_to_pointer(®->override_prel); + + for (f = 0; strlen(reg->fields[f].name); f++) { + u64 shift = reg->fields[f].shift; + u64 width = reg->fields[f].width ?: 4; u64 mask = GENMASK_ULL(shift + width - 1, shift); + bool (*filter)(u64 val); u64 v; - if (find_field(cmdline, regs[i], f, &v)) + if (find_field(cmdline, reg, f, &v)) continue; /* @@ -230,16 +256,16 @@ static void __init match_options(const char *cmdline) * it by setting the value to the all-ones while * clearing the mask... Yes, this is fragile. */ - if (regs[i]->fields[f].filter && - !regs[i]->fields[f].filter(v)) { - regs[i]->override->val |= mask; - regs[i]->override->mask &= ~mask; + filter = prel64_to_pointer(®->fields[f].filter_prel); + if (filter && !filter(v)) { + override->val |= mask; + override->mask &= ~mask; continue; } - regs[i]->override->val &= ~mask; - regs[i]->override->val |= (v << shift) & mask; - regs[i]->override->mask |= mask; + override->val &= ~mask; + override->val |= (v << shift) & mask; + override->mask |= mask; return; } @@ -313,11 +339,16 @@ void init_feature_override(u64 boot_status); asmlinkage void __init init_feature_override(u64 boot_status) { + struct arm64_ftr_override *override; + const struct ftr_set_desc *reg; int i; for (i = 0; i < ARRAY_SIZE(regs); i++) { - regs[i]->override->val = 0; - regs[i]->override->mask = 0; + reg = prel64_to_pointer(®s[i].reg_prel); + override = prel64_to_pointer(®->override_prel); + + override->val = 0; + override->mask = 0; } __boot_status = boot_status; @@ -325,8 +356,9 @@ asmlinkage void __init init_feature_override(u64 boot_status) parse_cmdline(); for (i = 0; i < ARRAY_SIZE(regs); i++) { - dcache_clean_inval_poc((unsigned long)regs[i]->override, - (unsigned long)regs[i]->override + - sizeof(*regs[i]->override)); + reg = prel64_to_pointer(®s[i].reg_prel); + override = prel64_to_pointer(®->override_prel); + dcache_clean_inval_poc((unsigned long)override, + (unsigned long)(override + 1)); } } From patchwork Fri Nov 24 10:18:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CDA9AC636D3 for ; Fri, 24 Nov 2023 11:31:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=77X0Ed32mZ93wkyZeolV+zPDRMoCn22EXz9ppp0UzPc=; b=lnUxVJbbWKZEjXf6o9b77Xexhd qRlM/E7eT6S8IfjF2d26jyGuJBIIJZcpO2kSK3ZVVMm4+iO5n7O28yhJxgTq7Xli7GHLXoQpR9E5B +aGWHUGLiR2g/jcSt62S+UFR8DkVy5NzZTdQ9YaZidPIzWN9tK845LAb9wWeeLSQsj9JUBJt8HlP0 xdGnlRL5R72F1dhVdr9A253YCSCVT3y+WV5Me3+n9WZUxp82KEUkaxS+KgurVJ1vEZAyfiCq+Ir0c +SY9gQEwTCQ1wZ6K5c7K2RXx82nDchrc1RfBgyY0xVSRKZAsDIg2MbKAeW2w6sQjgOwXUmOumPgxg wFrzCrzg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6UPV-006yT7-12; Fri, 24 Nov 2023 11:31:49 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIp-006ojN-2K for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:20:57 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-da307fb7752so2421366276.0 for ; Fri, 24 Nov 2023 02:20:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821250; x=1701426050; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=s88eJYP3maYPG/OYptXD7KAF+qfIselP+gc+NxV5N+4=; b=j8dTCN+dDK/+QbTg8gz1dy4dN2+dRu/bgbfcwyiPNBDqGp+lhrilMEezz3dybeZ7Tl nNjOBJV4BHeqheirx8VSFEtgu1UxqTggoD4lDxl517bP/5LY88WlbvsvNCYin61KRWlz V7LpRchzGj25WmaUUHbDp2lGQ20oznCw5LnGM/bYvYCJdAhR6jbZ+kLzJ1HtyoUmNkAG XHpn1Ih4e/R5Ya3BLLoX7/QXLEiXI9laO06wLVA1wYnq1T+6YfbU/m7MQh59mEVzLezh GCljoABgVyQDYhPpOsffc9DgzE1x2KEGqlhFVVYr+IGC9GQLzBRD0ZqrsRXxIgDURQ4i Jl3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821250; x=1701426050; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=s88eJYP3maYPG/OYptXD7KAF+qfIselP+gc+NxV5N+4=; b=byj6nrOws/x4VSDSz/VsQfOJzFU+/O8O89LOufY1cbboPKWVvOUiZ+aW240mi8TN4L M1nS7WeKW15GUdyqRrJW/cFDB5bVCOmVt6+9QuSvdekheMn5KxA8RuoFKJ4ZWt//r3A1 b/VZEJNq5UG3DfWdw0VDQCwk0dJlp2owUBhCN001hNoNJNovIYB1/IhiHWEK3bA3R1dl jOJvXtVUC+4aZ0/f90kgHGxQIPfhQOVq4UULCdmudtPnD1onFl6TwPHQ/GZDiur+RSah MnBU3ij77W9XR0Lz50BIHkNR6sf6iWno1bHha5OovlmTNFQVWDSeOZWEUy2FgUffNYKy eUFw== X-Gm-Message-State: AOJu0Yxvupu10zKn1Dq6vytH26mb8f6pfKUvaCQ7X6MYIPKOARwITfTt izYA9XsNC+gRzmseoVahLI9z2RM6LIL0HJysGCXeW92MGPKonwE+INlt2yw+qDIRWQjn1LgijOK KMyQu0FYdHBklhN+DNVdPPwVlKD9Q7BYMshky0Cx1vB3cDui9MVAKQ6TghLkWMS7ef61P7qQBkB c= X-Google-Smtp-Source: AGHT+IER/9IVqXVnzZ9fjYR2TKhwowaYhx2WeGIerc71ZqyOi79XWqG4iEbBhzMI0yLeK7AwP5uYkOis X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:d856:0:b0:dae:e8dc:c026 with SMTP id p83-20020a25d856000000b00daee8dcc026mr61027ybg.13.1700821250027; Fri, 24 Nov 2023 02:20:50 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:56 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2767; i=ardb@kernel.org; h=from:subject; bh=yiHvX4IJSuwnRIEhgsJMeOuKMVRgGaEj46ggQfAoQSM=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYvJHv6/8C2ZPZfxeMm3m1q/a117xVv37zSX6pqjm2 8eyiY5bO0pZGMQ4GGTFFFkEZv99t/P0RKla51myMHNYmUCGMHBxCsBEBBcwMmxWO91qai5TdY7f +RWH2SeJkrCijkRdiauzr1d/ntaxYRkjw6mZvzbOyp8T+CPOIy55V8MUyTXZSe8+beObtX+JQtE xLzYA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-57-ardb@google.com> Subject: [PATCH v5 16/39] arm64: idreg-override: Avoid parameq() and parameqn() From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022051_787989_76D54C39 X-CRM114-Status: GOOD ( 18.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The only way parameq() and parameqn() deviate from the ordinary string and memory routines is that they ignore the difference between dashes and underscores. Since we copy each command line argument into a buffer before passing it to parameq() and parameqn() numerous times, let's just convert all dashes to underscores just once, and update the alias array accordingly. This also helps reduce the dependency on kernel APIs that are no longer available once we move this code into the early mini C runtime. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 28 ++++++++++++-------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index 4e32a44560bf..cb4de31071d4 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -194,8 +194,8 @@ static const struct { char alias[FTR_ALIAS_NAME_LEN]; char feature[FTR_ALIAS_OPTION_LEN]; } aliases[] __initconst = { - { "kvm-arm.mode=nvhe", "id_aa64mmfr1.vh=0" }, - { "kvm-arm.mode=protected", "id_aa64mmfr1.vh=0" }, + { "kvm_arm.mode=nvhe", "id_aa64mmfr1.vh=0" }, + { "kvm_arm.mode=protected", "id_aa64mmfr1.vh=0" }, { "arm64.nosve", "id_aa64pfr0.sve=0" }, { "arm64.nosme", "id_aa64pfr1.sme=0" }, { "arm64.nobti", "id_aa64pfr1.bt=0" }, @@ -224,7 +224,7 @@ static int __init find_field(const char *cmdline, len = snprintf(opt, ARRAY_SIZE(opt), "%s.%s=", reg->name, reg->fields[f].name); - if (!parameqn(cmdline, opt, len)) + if (memcmp(cmdline, opt, len)) return -1; return kstrtou64(cmdline + len, 0, v); @@ -281,23 +281,29 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) cmdline = skip_spaces(cmdline); - for (len = 0; cmdline[len] && !isspace(cmdline[len]); len++); - if (!len) + /* terminate on "--" appearing on the command line by itself */ + if (cmdline[0] == '-' && cmdline[1] == '-' && isspace(cmdline[2])) return; - len = min(len, ARRAY_SIZE(buf) - 1); - memcpy(buf, cmdline, len); - buf[len] = '\0'; - - if (strcmp(buf, "--") == 0) + for (len = 0; cmdline[len] && !isspace(cmdline[len]); len++) { + if (len >= sizeof(buf) - 1) + break; + if (cmdline[len] == '-') + buf[len] = '_'; + else + buf[len] = cmdline[len]; + } + if (!len) return; + buf[len] = 0; + cmdline += len; match_options(buf); for (i = 0; parse_aliases && i < ARRAY_SIZE(aliases); i++) - if (parameq(buf, aliases[i].alias)) + if (!memcmp(buf, aliases[i].alias, len + 1)) __parse_cmdline(aliases[i].feature, false); } while (1); } From patchwork Fri Nov 24 10:18:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467570 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9D162C61DF4 for ; Fri, 24 Nov 2023 11:31:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=1NykBQR8lpEJeOF5CUaISm5nV/fUOFf/XjFfUxJBgaA=; b=yJx6IOXLTb2BhQIj1cGTSJP6bX YVk3wq4KltYYE94SVwry4KD8OT1jaPwaF1ZTWhJG2RG5BkjJxrkh/yaplEM7j1Z8q/GME1Sf2ja4p 02wWCUrg2G0hmdIeACqfEwvzBMIgnbeOtUqapg5hzpzhKFBbd82IjfrnnjDr3QZKahXsDufzNdn1w Ttq5YsmqbfzbOvMvtecrBS+gqF0pJyNX7yadjQ9R0doRZdsPAzO1KcNvi7d3CImm6lLVeelWq8BsC X65UbMuAhedBKCqoS8jPnfbw/KHg/SmFsmqgmUCnbzHG2TkvYbSFCB+SoZKOVqOeSAyiwC57ZCqrT 7GtKuF0A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6UPW-006yTq-1N; Fri, 24 Nov 2023 11:31:50 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIr-006ol4-33 for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:20:59 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-daf6c1591d5so1320468276.1 for ; Fri, 24 Nov 2023 02:20:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821252; x=1701426052; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QtqbrIoySFcqw8IOnirpKEanKaAyPmD2uRlCTeIOgDE=; b=k0K38BWZXyKnpKrd2lFHl0/5W9/601YFosjtqN5uKleyvQIIksX1E1Z1nG4r9QU2Rs TA+OU2Vk989CM4fb7My3bAuwVWivuoi4OtkGbYul1JMC+Tws6hgS9/i5Yubb4YaLe1Rs 9PKMzFluMY0lwzKnSF+zWaK4oYZ1GQLGVgauoRuKaaA5crI4qjnkYvCyLO9zxyxp+j6k vGLyDTkFCmDtr1kIsSitsUHvmdPgrf0glSDgvmHPIzXszttERqZJkyZWYKaHDDzeVdhY xMtlBy1bdjsgojiaO8ZIz8fJ57nkueyi6ydbDRzmLfNSFY6q9VlzQdICPLTQedZjyJdR HTMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821252; x=1701426052; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QtqbrIoySFcqw8IOnirpKEanKaAyPmD2uRlCTeIOgDE=; b=PujjdIylCfaNDjNa9Xcju11QMr/2l6iJgnERrKJ1G6/gl17IqxJeYQFN6H6Ufp0DVc SppQ+9KPtCNsV2VbFemMI/Pg2JOf3LCCAD+lZHwKhulobobodX8sB9BS3JvcQMjqd15P Mv9f/qIuNTHIzBOS0OF5dmciqfAQ5lWILuWZMB1Jk0fLPJL4sSZq1DqbDObo77Fhlz7q CIJGMbC470SCXuwv1lj9m+D3+fid04Bs5a9XvsNBKFSmSTsd0CsW4hiFpE3aZTQAA9sI KUqjWaFvlyACrmQW9eS854dH9kJYEbmvSBIA+ElmCgeA844io7n+9VDrKYDgYyVetxn7 4t9g== X-Gm-Message-State: AOJu0YwEWTsxNQ9fW4KFv0YVSSZ7dxjuqPoqfpDQwt75nb6GPewWLf2a 6WWXV/eyIoNMrjxG1hw5mD0fZ4PjK1y+HZeJ2eLRYn9iYUd3Po3zrW7wrqezSSCEXj5B8oQg19h DClYuHxBzUcnpfgthug1lngpcmw3RXvhqoPtHx5WLWIRS3HWfaIeV1CQdChM627BjV0mttsDwcp M= X-Google-Smtp-Source: AGHT+IHueLnVKYBU3w6X7inJD1sa95OAnuKUXMIEtFknELJcsqt+xrLm1AmGs5Epthe17usHy5UyLsgB X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:6c89:0:b0:da0:3bea:cdc7 with SMTP id h131-20020a256c89000000b00da03beacdc7mr67176ybc.2.1700821252153; Fri, 24 Nov 2023 02:20:52 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:57 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=928; i=ardb@kernel.org; h=from:subject; bh=HZ7unQ26Ahuxg+B9w+1yOuMRfixVzArffI1/mgiVf/w=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYsrMPf+rJXKevbpY9Pbjk8M/XJ8u1GRJOhFSxP28P qVj+pPCjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRdYsYGbrZs/WC3ud6nVCe d6FDKDVO3eBt99ekDXVr7/93Wca4aR3DP6ufb54ycVXffhn874OV+/VLIratphNb9toLT1MX+GH 2iwcA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-58-ardb@google.com> Subject: [PATCH v5 17/39] arm64: idreg-override: avoid strlen() to check for empty strings From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022054_017996_D554B578 X-CRM114-Status: GOOD ( 13.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel strlen() is a costly way to decide whether a string is empty, as in that case, the first character will be NUL so we can check for that directly. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index cb4de31071d4..96fec5b7a65e 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -241,7 +241,7 @@ static void __init match_options(const char *cmdline) override = prel64_to_pointer(®->override_prel); - for (f = 0; strlen(reg->fields[f].name); f++) { + for (f = 0; reg->fields[f].name[0] != '\0'; f++) { u64 shift = reg->fields[f].shift; u64 width = reg->fields[f].width ?: 4; u64 mask = GENMASK_ULL(shift + width - 1, shift); From patchwork Fri Nov 24 10:18:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467568 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 91BC6C61D97 for ; Fri, 24 Nov 2023 11:31:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Zbirus4uWFMFyTzxL6oZTZELWjzRpJSP4/yKBWC6gFk=; b=Bn93FxOeHa/Xj3CfakhHYy74jy VuHDoSdacak8Ybf5HjRw4EDmC64pdrkDi0P0c18RYki6CAC42C+qBIEbutYMO7STZlQp/DgyuQ2NG NbknX0a++qrDsXEst/uLdH0Czf64+ffCzklTJWxvIJWX+f1lulD3+kDYNR6IDU4jeGwmwEzXFPizH EcpctE4NrjU6SR0+3lJDLDudzBIxCVu0Yhk2tyQtpX5F2yC5QewQCm73jnPp+/SapXXxS1lQv95Lf cLNjGT/5sNccCiiG1pm77K05ggGLTU+aedLdIgouCcRjGRGjnaIeDDK87sfW2LYKLXif97swO+haU XBrMOWAA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6UPX-006yUm-1S; Fri, 24 Nov 2023 11:31:51 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIx-006omk-0Q for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:03 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5ccf5d22873so26882247b3.1 for ; Fri, 24 Nov 2023 02:20:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821255; x=1701426055; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=l5ZZZnEoSFivql5UHpCJhbC6GFrfdxCztP0ALxEGa0c=; b=LiJgNhAtYMFTwngPKPza4kqe69ZIILPGfZ7p6DmboOJ1dVK3N5vLOo7eJvtU4Q6pCX e3pSSROJV7Hvho+ijSLYAt9/9zSJYsfKx1VwTcfz0dhQpLX4O6PqETfJUa358FAA1EtY xiJW14d8DQrsxZ/+h6eX4Mh/00BGjcf8n7mxWBhfOQEfEzOMyfpy8BchfVp6vs3wHT71 bfitQwqDDgpAWNQPtoMasUinqstquPeJyV/PYvf5Rg7cXVWkFBZ/s9QiMFlbYdQvUGNR C/pbyE4QxA2mbaahxzCSw2Ubo6H+R5QvM6grGRCBMa6P+DGFQOSKi+7SsP/HAm8YiiYC c1Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821255; x=1701426055; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=l5ZZZnEoSFivql5UHpCJhbC6GFrfdxCztP0ALxEGa0c=; b=ElEMAER57sM2b9xi7x5RLqV/xvKtEYB+ohCgqXhcK4hFxHRXLDGennC+fNhn5tmmrG VfbJL/6rd++v5mzJaWZ3R1CFJpi+uNkJr66Bz/xZNLKRomf+GCnyhwmzWpyl6HqAW45S 8PwD2F/VSNKOyhyfY+2oR2HEB78D2jHFNC5ujIB/MWohGblSvvUvHygrRyUJap6zGYBa 6VQpWUjFQA/W8zKNLOTbHXoQ3eii2I6xSRTOuvmwVifWsaP/GcMfyhC4s6MocXSvKbZW HHqCSqTspUk3lkXxYhgipfaeXZjM8F74uH5vfuBVhLGK1y/nmPjAPP5T021JjrDOg88T b45w== X-Gm-Message-State: AOJu0YyWV/2bivnCagsEA/qvyzY4+uPk9mWXEzR2CFqI47c+SuAUDfIc akQYBhEAZALd75ih/S+A0SUDkErmw3OvB0CSRqXIK+2qc8ylnEJ4LnoV2kZjqX8y+xofMlNFNFB fIWzdt/7rvG/bS6ZJ41H+U5Q4yUCbyR9DBRc6ZO+p8tVf/BQG41gnJxaOEQHSeJlTwnE2nvH4OK s= X-Google-Smtp-Source: AGHT+IHc3GegYcnvfs6Zy5Nk95WyX9va0N2plqPGDWk8KBWsNceQoq83WqlwSdY8FLpEBoYWxIBHI08N X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:690c:845:b0:5a8:205e:1f27 with SMTP id bz5-20020a05690c084500b005a8205e1f27mr69580ywb.6.1700821254756; Fri, 24 Nov 2023 02:20:54 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:58 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2434; i=ardb@kernel.org; h=from:subject; bh=lC4DkQ4BnH8UW4k/1dPlaimYll+z9uySAutwLfWCeZg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYtqJ6/PNZn6a9/DTL6Hrx5rd192oPhZspLxZOGl9Z tFdP83fHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiC2cwMpzabm4ef3iSgH3f 10W68YHviguu3ujStfTbctLjEcPXxnCGv1L7VL4aRyW4GUR4TBI0eazAsKrpjsLu0tkWqZvYQyc oMAAA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-59-ardb@google.com> Subject: [PATCH v5 18/39] arm64: idreg-override: Avoid sprintf() for simple string concatenation From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022059_198712_315688FC X-CRM114-Status: GOOD ( 16.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Instead of using sprintf() with the "%s.%s=" format, where the first string argument is always the same in the inner loop of match_options(), use simple memcpy() for string concatenation, and move the first copy to the outer loop. This removes the dependency on sprintf(), which will be difficult to fulfil when we move this code into the early mini C runtime. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index 96fec5b7a65e..41eeca857b32 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -215,14 +215,15 @@ static int __init parse_nokaslr(char *unused) } early_param("nokaslr", parse_nokaslr); -static int __init find_field(const char *cmdline, +static int __init find_field(const char *cmdline, char *opt, int len, const struct ftr_set_desc *reg, int f, u64 *v) { - char opt[FTR_DESC_NAME_LEN + FTR_DESC_FIELD_LEN + 2]; - int len; + int flen = strlen(reg->fields[f].name); - len = snprintf(opt, ARRAY_SIZE(opt), "%s.%s=", - reg->name, reg->fields[f].name); + // append '=' to obtain '.=' + memcpy(opt + len, reg->fields[f].name, flen); + len += flen; + opt[len++] = '='; if (memcmp(cmdline, opt, len)) return -1; @@ -232,15 +233,21 @@ static int __init find_field(const char *cmdline, static void __init match_options(const char *cmdline) { + char opt[FTR_DESC_NAME_LEN + FTR_DESC_FIELD_LEN + 2]; int i; for (i = 0; i < ARRAY_SIZE(regs); i++) { const struct ftr_set_desc *reg = prel64_to_pointer(®s[i].reg_prel); struct arm64_ftr_override *override; + int len = strlen(reg->name); int f; override = prel64_to_pointer(®->override_prel); + // set opt[] to '.' + memcpy(opt, reg->name, len); + opt[len++] = '.'; + for (f = 0; reg->fields[f].name[0] != '\0'; f++) { u64 shift = reg->fields[f].shift; u64 width = reg->fields[f].width ?: 4; @@ -248,7 +255,7 @@ static void __init match_options(const char *cmdline) bool (*filter)(u64 val); u64 v; - if (find_field(cmdline, reg, f, &v)) + if (find_field(cmdline, opt, len, reg, f, &v)) continue; /* From patchwork Fri Nov 24 10:18:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467569 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4134FC636D4 for ; Fri, 24 Nov 2023 11:31:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=V7dr1lky53nmQ4lnKWgSQ11ZO97NRSJw7hOHfU0PcZQ=; b=a9hokLFYYrZJesFkVSGDvENSkv hpuWFjwUbPafXJ8zPszIhvy3/OX7qeZi6ACHhFx5T78+0IKXmjBZZdbH2T4CVcCz7dvsXJkBmeB9h 09Mm17dGlUcFh6EqmrYX6Wg8+m7qbd9hWLe8LEd0av/AJpCK41DR6+OkWn7g1GhMKWvfP/2UpK9p/ fVkMHRmVrWa6qyNJntCY+K1UOk2VZieRRZUnaE8Q7hURd0/YvYUux9I2eo3ilO5ptYIaSjzl+Vb/w T+70SsNDg5JxUX73zwGGea31nVb9VbW5uyCidU1aXDSFo2xbxf4buhbNkorJc9+grQLgq/QW3uM8W wRKjHfqw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6UPW-006yUC-33; Fri, 24 Nov 2023 11:31:50 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIx-006ooG-0G for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:01 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-da0ccfc4fc8so2405371276.2 for ; Fri, 24 Nov 2023 02:20:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821257; x=1701426057; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zqx2vN/ooTrKAVWBshy8Ta+sP+wPrsjx5cWvmkNx64E=; b=1ua7YKmb6nKaJP2hy5ovc0rbtzP60QIJZbhWe+zNfD6+/BJpLfHAzmZMcSV+SemsIi XAXnAPV0ZUE1XmXfaBzvLrcjb28OB71ZRj6uOQ1A6qOn0guT+u33CWH/OGpsZcCN5lUb OxfuYRoH20bBLe674/1qowUUUdxzQvG4j8gDfOBaKUp1C7YxTon9hIZDt8hSS7Kd/IGK ui+jzb3L01FRQIrD1tikIEqeYieo7oR10jyTC+jP2JFLlRLMz1BzolOpTzhFxJoAgZO4 G9sfL06dHjelU/KCXjLIsVKnZH8P6OvRnFgVZbAO40MU9ml9VV7p2wnLHRA+QUoOe9ue dSqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821257; x=1701426057; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zqx2vN/ooTrKAVWBshy8Ta+sP+wPrsjx5cWvmkNx64E=; b=OGNupzBUauU8IEfH5BA++064ij1JmNdvC+S9jWH7KcjNgNSjGsxApoQtYtMj5r6TbF HSaOW6PP7QmmwiauKYeMnqJiR+g1wMQFqQl8fklwaQmpSFV2PamLrYse+df1r6kmk+z8 3Qpk0737Pixrr+07rUd/+GodUKxkq9o0EskW5FlGFfdwtQB+mqQEH34nBtWblUYHbyFT dPiTDMfHqiW65fzcCh2przK467K42kcPDGWDX862E5kCyWibRvT4ams9Y1eHv8I4GWUb GaqmMUyvrU8ACx0q38De5dmreFjaF3hlQ8dCTrfW7R9XPF23bf03uQ7QD5HjtOWVdsI5 gTdA== X-Gm-Message-State: AOJu0YxRhjhBjxExUZACl+agayHtjEIXB9YZuqeiEX9n4HIEScl/167J o3NBz4BERrcKAQLU0fjzV14Y8LItHqhJxHbudZnO1zraAKNh8eFK19ihfPfIujXdrvOPnM4cbyn svbULRruxQBOO/whfyd/KpAKlhiVgTG3+Qw0M2Syzpw97QCZ3ShkXRrUI1bon75pxYG77QTA/Z+ M= X-Google-Smtp-Source: AGHT+IHZKgRwMYQZK6nTreuKpHl3uVJzY2b7wu0oTmBeD0TQGhq389hf+I4fEpFwnrqD2OBTMab6QO0E X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:cf14:0:b0:daf:660e:9bdb with SMTP id f20-20020a25cf14000000b00daf660e9bdbmr65973ybg.6.1700821257141; Fri, 24 Nov 2023 02:20:57 -0800 (PST) Date: Fri, 24 Nov 2023 11:18:59 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1683; i=ardb@kernel.org; h=from:subject; bh=Px7/KCxFjrRFAVEoUqGQ547s7n386NOxqHmoPGXvkvQ=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYsbkFb050jWLOZNPnKw883c/79m3G41eu0cx10+fb pX8Odmlo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAExEKYbhn45OT/6OrF1136LC lpg6ey/7JVe1mbntusjK6w9ntISLRTP8U94RLL7QZfqhfuFqXS3OWt2Nvv+Yn87Vf7JpxSGF8Pi t7AA= X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-60-ardb@google.com> Subject: [PATCH v5 19/39] arm64: idreg-override: Avoid kstrtou64() to parse a single hex digit From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022059_133896_46F028BC X-CRM114-Status: GOOD ( 14.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel All ID register value overrides are =0 with the exception of the nokaslr pseudo feature which uses =1. In order to remove the dependency on kstrtou64(), which is part of the core kernel and no longer usable once we move idreg-override into the early mini C runtime, let's just parse a single hex digit (with optional leading 0x) and set the output value accordingly. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index 41eeca857b32..2eb81795934a 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -215,6 +215,20 @@ static int __init parse_nokaslr(char *unused) } early_param("nokaslr", parse_nokaslr); +static int __init parse_hexdigit(const char *p, u64 *v) +{ + // skip "0x" if it comes next + if (p[0] == '0' && tolower(p[1]) == 'x') + p += 2; + + // check whether the RHS is a single hex digit + if (!isxdigit(p[0]) || (p[1] && !isspace(p[1]))) + return -EINVAL; + + *v = tolower(*p) - (isdigit(*p) ? '0' : 'a' - 10); + return 0; +} + static int __init find_field(const char *cmdline, char *opt, int len, const struct ftr_set_desc *reg, int f, u64 *v) { @@ -228,7 +242,7 @@ static int __init find_field(const char *cmdline, char *opt, int len, if (memcmp(cmdline, opt, len)) return -1; - return kstrtou64(cmdline + len, 0, v); + return parse_hexdigit(cmdline + len, v); } static void __init match_options(const char *cmdline) From patchwork Fri Nov 24 10:19:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467448 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A349EC61D97 for ; Fri, 24 Nov 2023 10:22:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=NcDszQsMpkUBs4PlRrWsEe9Ih2M3MpfbtAdRKGqonuY=; b=DpLNi4YtHdmGOegcuxuHscJsJm AmekuImycjbT/+IJWuHJS0V8CGJuJu4+C422VEEDktKJLI5ZJ8ZlNlQlN9JhyNBVKZV8QYWA7uWbr 8wpQ0cy6cFYb3yLE+qBWctuO4L0QgZyLj2wSa0G2LgyhNuxdposClAmbLrgLyQPE/8sUyCfZtQDYK N8Qr54q9MLpUNxRjm44r+t+GGhVQ17Qy4hbnHfmfAoLbRb+LBoUsB52Gd2tCvByx31pluJc/ZL/JE vuyHFPribSs43lxTXrpHvPNfu4CuIOlk/tMrE7+DBqgMtHxVPT8fafJA83ISEys8/lUlS8X+Lezt9 Q7VvNaGg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJV-006pEg-1j; Fri, 24 Nov 2023 10:21:33 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TIy-006oqB-37 for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:04 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5cd1172b815so10912567b3.0 for ; Fri, 24 Nov 2023 02:21:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821259; x=1701426059; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=vRjIhLF/c02dlJNC4AveLv6X89IoKzOMf0jmwu6wZWo=; b=q75S/pjMEB3m2J+51h7++CmWRAKYsaDEIEKFXIsLijVz+PsxEOAE5BX09zy7bZtGzo 49tfK7/CIfvS+ZiQco7qzxV44ffbUxkavBAraSVZvaoSwJNb1+Mux8HxaQLUsA4aFPl/ gD/wM99m84Ai3xhqoNydpvu4fZZ/qtC8sg1Q6+i72PfFVK6a7umZxoMILKv6an+ueASU /RC7QMyCQ/W6OGrZpip6Xp4GAyR2MGfWTBwT/j4vpD7G9o4Yxvc00Gpwa30OxJGvwCau LxzYVGPVIYfDrUe6/S6j4VihZ2JdDAdCEdzT6H28JQWtrHTCWb35maNEkNR+uBAu6RiZ W3jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821259; x=1701426059; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vRjIhLF/c02dlJNC4AveLv6X89IoKzOMf0jmwu6wZWo=; b=JrYDO/xXSqjRJ4E1eEw8VmjW6n7sYFCwVsfQCep6Voe6AMXFv9vL49RXYqa3/qOUh5 AzTFPYNOkhms9WfYBZauoid9hH5oF4Wf7QgTElAuPqFLmANvunv0d9443ZfXYDo5283c 4Ofdtpcutgx3unj1vwLWMW9GpTrIGkixTCmpNwsGs0iRJV/wj6XhkaojE6OPn1DEWWVI UxINIBBbBJsCEApaxUbjRtBnI3p0xASHaZ5CijW9PyEF7NdtI6ridmurRcoZjZ3tRO4z vpGL9pHFMlphjaLioTJ9oZy7pSBIB/jNrgCr/lLcNcpoegItaOlRPojS8AG6baTb5GtG RKLw== X-Gm-Message-State: AOJu0YxmjFkbk04qSs2MFNzeLWBV47wCvJnPbjYVWeomjOZrFZqe2TI/ U3iM00SkUergWw1aRqTt5ZzkVDnJPtlvGstHvvfZ3zEZ1B1nKTMeAsrBFU6RBRaLu4AOYZiNJd4 iKHDUpK/3gYbO08CQPvvuvZINHEhi68YfGZUG4kFTJARIv7YgEH9xVFi5NzICZc2lZk72VuC/u4 4= X-Google-Smtp-Source: AGHT+IGIGWTphxAo0Kg9hxqJtc1OQnJysmDG8imhrqKx8VE2PPWZQlT4fXFC9zGaDHvWurcOvpan3tYF X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:690c:2b82:b0:5cb:ed68:a256 with SMTP id en2-20020a05690c2b8200b005cbed68a256mr69642ywb.4.1700821259512; Fri, 24 Nov 2023 02:20:59 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:00 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6735; i=ardb@kernel.org; h=from:subject; bh=n5mlfNOnrh0W3vmiJVYYcAYcikQNxSumvqAO9oGdbII=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYtY1JrOVvpcKTz17sXzDPlkZHc9JXVMWxm4pjTd7q aqWMmFzRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjI8hMM/+sO2M4/eamKa29j 2/3o+eaP5AOe+WnJZ5vOjFdiEjshb8bIcP+t5f7tgtvkvhbnuOV9YlyW77Doxiq7qGeisnqZ/67 e4gIA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-61-ardb@google.com> Subject: [PATCH v5 20/39] arm64: idreg-override: Move to early mini C runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022101_038593_FE75B32F X-CRM114-Status: GOOD ( 17.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We will want to parse the ID register overrides even earlier, so that we can take them into account before creating the kernel mapping. So migrate the code and make it work in the context of the early C runtime. We will move the invocation to an earlier stage in a subsequent patch. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/Makefile | 4 +-- arch/arm64/kernel/head.S | 5 ++- arch/arm64/kernel/image-vars.h | 9 +++++ arch/arm64/kernel/pi/Makefile | 5 +-- arch/arm64/kernel/{ => pi}/idreg-override.c | 38 ++++++++------------ 5 files changed, 30 insertions(+), 31 deletions(-) diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index a8487749cabd..dc85dc2ee4ed 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -33,8 +33,7 @@ obj-y := debug-monitors.o entry.o irq.o fpsimd.o \ return_address.o cpuinfo.o cpu_errata.o \ cpufeature.o alternative.o cacheinfo.o \ smp.o smp_spin_table.o topology.o smccc-call.o \ - syscall.o proton-pack.o idreg-override.o idle.o \ - patching.o + syscall.o proton-pack.o idle.o patching.o pi/ obj-$(CONFIG_COMPAT) += sys32.o signal32.o \ sys_compat.o @@ -57,7 +56,6 @@ obj-$(CONFIG_ACPI) += acpi.o obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL) += acpi_parking_protocol.o obj-$(CONFIG_PARAVIRT) += paravirt.o -obj-$(CONFIG_RELOCATABLE) += pi/ obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_ELF_CORE) += elfcore.o diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index a8fa64fc30d7..ca5e5fbefcd3 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -510,10 +510,9 @@ SYM_FUNC_START_LOCAL(__primary_switched) #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif - mov x0, x21 // pass FDT address in x0 - bl early_fdt_map // Try mapping the FDT early mov x0, x20 // pass the full boot status - bl init_feature_override // Parse cpu feature overrides + mov x1, x22 // pass the low FDT mapping + bl __pi_init_feature_override // Parse cpu feature overrides #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS bl scs_patch_vmlinux #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index e931ce078a00..eacc3d167733 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -37,6 +37,15 @@ PROVIDE(__pi___memmove = __pi_memmove); PROVIDE(__pi___memset = __pi_memset); PROVIDE(__pi_vabits_actual = vabits_actual); +PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); +PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); +PROVIDE(__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); +PROVIDE(__pi_id_aa64pfr0_override = id_aa64pfr0_override); +PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override); +PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); +PROVIDE(__pi_id_aa64zfr0_override = id_aa64zfr0_override); +PROVIDE(__pi_arm64_sw_feature_override = arm64_sw_feature_override); +PROVIDE(__pi__ctype = _ctype); #ifdef CONFIG_KVM diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index d084c1dcf416..7f6dfce893c3 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,6 +38,7 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := relocate.pi.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +obj-y := idreg-override.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +obj-$(CONFIG_RELOCATABLE) += relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c similarity index 94% rename from arch/arm64/kernel/idreg-override.c rename to arch/arm64/kernel/pi/idreg-override.c index 2eb81795934a..bcba0ce71af0 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -14,6 +14,8 @@ #include #include +#include "pi.h" + #define FTR_DESC_NAME_LEN 20 #define FTR_DESC_FIELD_LEN 10 #define FTR_ALIAS_NAME_LEN 30 @@ -21,18 +23,6 @@ static u64 __boot_status __initdata; -// temporary __prel64 related definitions -// to be removed when this code is moved under pi/ - -#define __prel64_initconst __initconst - -typedef void *prel64_t; - -static void *prel64_to_pointer(const prel64_t *p) -{ - return *p; -} - struct ftr_set_desc { char name[FTR_DESC_NAME_LEN]; union { @@ -329,16 +319,11 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) } while (1); } -static __init const u8 *get_bootargs_cmdline(void) +static __init const u8 *get_bootargs_cmdline(const void *fdt) { const u8 *prop; - void *fdt; int node; - fdt = get_early_fdt_ptr(); - if (!fdt) - return NULL; - node = fdt_path_offset(fdt, "/chosen"); if (node < 0) return NULL; @@ -350,9 +335,9 @@ static __init const u8 *get_bootargs_cmdline(void) return strlen(prop) ? prop : NULL; } -static __init void parse_cmdline(void) +static __init void parse_cmdline(const void *fdt) { - const u8 *prop = get_bootargs_cmdline(); + const u8 *prop = get_bootargs_cmdline(fdt); if (IS_ENABLED(CONFIG_CMDLINE_FORCE) || !prop) __parse_cmdline(CONFIG_CMDLINE, true); @@ -362,9 +347,9 @@ static __init void parse_cmdline(void) } /* Keep checkers quiet */ -void init_feature_override(u64 boot_status); +void init_feature_override(u64 boot_status, const void *fdt); -asmlinkage void __init init_feature_override(u64 boot_status) +asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) { struct arm64_ftr_override *override; const struct ftr_set_desc *reg; @@ -380,7 +365,7 @@ asmlinkage void __init init_feature_override(u64 boot_status) __boot_status = boot_status; - parse_cmdline(); + parse_cmdline(fdt); for (i = 0; i < ARRAY_SIZE(regs); i++) { reg = prel64_to_pointer(®s[i].reg_prel); @@ -389,3 +374,10 @@ asmlinkage void __init init_feature_override(u64 boot_status) (unsigned long)(override + 1)); } } + +char * __init skip_spaces(const char *str) +{ + while (isspace(*str)) + ++str; + return (char *)str; +} From patchwork Fri Nov 24 10:19:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EBFA9C61DF4 for ; Fri, 24 Nov 2023 10:22:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=oPMQTCdgF1bpLyzCMw1DyVSg+NtNZw0OwrN50Ioc1T0=; b=DjYzYGXs7m14DCHZFfcLGy8Unx MW+elhGxQ5vpvfTRVJBzDkqqjj8oyRY+FYXGhyFHgDYd/0o8JiYsxEWxnYBpVt7TbOss48ViyNoWE /E61IepFzFN0aNM2kXD11OUDtv09UDoMkULp0a0Aqx5jlmroFmH1CxYpBn97t/w/8KCI3NvVNVLlU QbZaCgaGEnFGrZ31qmwLtmMpnEjK0zxVmFwoVG54SybintY8IsbgS7GknZexzJ0dS8aGtRXSh+CLN tj87DwbKEmWUy6KYl6P8tl7E3HZyJTf+tUtWsFpw3JjIIOkCjDaTBms1/h3KwIMWatf70ogqvcLIv T7kxqzMA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJW-006pFd-2D; Fri, 24 Nov 2023 10:21:34 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJ1-006ord-1z for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:12 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-40b394c381bso4093425e9.0 for ; Fri, 24 Nov 2023 02:21:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821261; x=1701426061; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=oxDiGbeLLIkW2GMFJY+N2ymcoPjact0Q84sjZzUT7ew=; b=JZ0kFBE0/OVWUGM2TNYtuq/aFuQXikA+x3G4Hyf/38qEijueESfalfsco77lBL5eTW XDvDAv/+Ko9vlAhJur54PqXGn7fB86THSi9RdCguHstDhbjNAJQ+z1mtNaHPgOpiwuod P4kPCxnVZlbl31sQ88ciyNTOBHN3zEZ+BI9GjM0gKgCUcaZ/8qxkN+aKwBqt8LXUXsZ4 zmxpqtFdrn+wShJ5G5BO9Sb+cnrmxz+3kj8Xe36b+d6vEb8oiGrmnrvBFiYkfnhwTvrr fh45VOLm2FAVPh5tT0Hp6YR85569IVLDWbENbUq5Oyt+7ZbK/22qFUbdAylkEwubKem8 X5jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821261; x=1701426061; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=oxDiGbeLLIkW2GMFJY+N2ymcoPjact0Q84sjZzUT7ew=; b=YWdhTM05D1negLUBk7zP2i4Y5Raux6+sBL1yxScxzi1y+3mXqYkTae1yLClvpep7jE xJSaefhxX0aLGgPvYAwZl6QgMQpFdo1CdpkR38WETEuycBi41f4lV08jgiUCBzra5GSD BawAOUSsFOuj1XkomfNtgsrDlrCtBO2yh6YK8+V8f7GtY3/OoJxY/Tkqc9EC3FmlqCVr plo6uVObpSM2LEoTgzVjSoJzUseGwWMHTuhplxldf8gMJKadR/HMLOozajBOJ6KS2rDq HiqWrAzHS6YmYVVMH55SU7P4Jf8KX4RaFNOLHTye6tzWSkXenJvqSMZgmrB23JFgDle+ fgZQ== X-Gm-Message-State: AOJu0YxobFhMvzYTXNEc3uJ+q7d6TOA8kUy8Qdsr7BUfu15iNRGbzDLh SfjQ4rzFdr6Dx1TLsTrc4hB7J+OKyDf3dMqj8gxfWxkERarSIxBqbYruOdoH54/OthpHvMcwHUL Y6QwVYCFC64kGDWiIlgJPWG9sXS8yVz1Y8dZHwxuDXAvudaCNhi49xePe/UonBKe+X3aqslv+1L U= X-Google-Smtp-Source: AGHT+IFzeeHZAKxkcEepWIzgokLWVYtOTk88jEUr6U33trGg7WaWswfzsIFHul5a7UbAttK8jzRi9nJG X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:4e4c:b0:402:f8e0:c500 with SMTP id e12-20020a05600c4e4c00b00402f8e0c500mr48692wmq.2.1700821261589; Fri, 24 Nov 2023 02:21:01 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:01 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1358; i=ardb@kernel.org; h=from:subject; bh=wxO7Rnsy3xuo0JEciNY7Qi3ys+7It4rA33719j9/65I=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYk4LA++Ss9NX+RmycjGxnSxuX7uctUlyWbix8O6FP 1hXhgd1lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgIk8d2RkeKt8gOeTZ7HC+6w/ h2OzLeeZ1ipHLbhid2DeWfGbM2qf3WFkuN+7vz7mp/FteeH3080fiVU16r/lsHJqY4zzOLpWbqM VKwA= X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-62-ardb@google.com> Subject: [PATCH v5 21/39] arm64: kernel: Remove early fdt remap code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022103_742077_3BAEBA46 X-CRM114-Status: UNSURE ( 9.97 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The early FDT remap code is no longer used so let's drop it. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/setup.h | 3 --- arch/arm64/kernel/setup.c | 15 --------------- 2 files changed, 18 deletions(-) diff --git a/arch/arm64/include/asm/setup.h b/arch/arm64/include/asm/setup.h index f4af547ef54c..acc5e00bf3b0 100644 --- a/arch/arm64/include/asm/setup.h +++ b/arch/arm64/include/asm/setup.h @@ -7,9 +7,6 @@ #include -void *get_early_fdt_ptr(void); -void early_fdt_map(u64 dt_phys); - /* * These two variables are used in the head.S file. */ diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 417a8a86b2db..4b0b3515ee20 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -166,21 +166,6 @@ static void __init smp_build_mpidr_hash(void) pr_warn("Large number of MPIDR hash buckets detected\n"); } -static void *early_fdt_ptr __initdata; - -void __init *get_early_fdt_ptr(void) -{ - return early_fdt_ptr; -} - -asmlinkage void __init early_fdt_map(u64 dt_phys) -{ - int fdt_size; - - early_fixmap_init(); - early_fdt_ptr = fixmap_remap_fdt(dt_phys, &fdt_size, PAGE_KERNEL); -} - static void __init setup_machine_fdt(phys_addr_t dt_phys) { int size; From patchwork Fri Nov 24 10:19:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467572 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B2363C636CB for ; Fri, 24 Nov 2023 11:31:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=KQa/ljbJ48ppHIMthzqbM3RhLhiBc0c5h3xfDRWz8Qs=; b=EJDJ93K0IjWxjCqM3I+KPse4hz eZOEwHcna0K+vrBQqlS4GwQN2QvlG68sRBPXfVZPPY1UVrplc7CF4lJlZUFDZsUk9RkW2AAL1Yedh xg/ua6gZHzhIYJIysmKu4EB/kz4itYuKbh8gKjIVtIXtNMHvpyOnhFufLdEyAOfaH9YR9mDG51g7N J57fjA9t26p60m5rwj0B6Eg63dCejchJxrgSWEW+QbSO7ZimJwc/p5eXGlp59CgW7Za1J0GtGW6gJ fww1jUcN3pAEtxSDfY7ZBmcQWdDBCobDYxrNOTTJOJaXyygWSkZ/F+ei0k0uO+FhxQo9YZKu/+S2d PRj1zIkQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6UPY-006yVq-2I; Fri, 24 Nov 2023 11:31:52 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJ4-006otv-0Q for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:17 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-40b2977d6baso8983825e9.2 for ; Fri, 24 Nov 2023 02:21:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821264; x=1701426064; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zO78n1C+IUUd4qQDduar+mDDdO70FT4CA4aVaSqaf3o=; b=pxsXNw8cpaYHd/KuLKIvW+/BUQYEnjGNA1bP19SwPWRJvU4x+Pf9HTjjZO6SHhViYb Xtuhs1l4PCY+LUP7aRrajDFaeK0Pgwv/ejhxFGGfehd7/rb1lLpxMGGoQ4TKgFuWlDp7 oWKKUSoG2W2QuV7gTZp4gdbvpyRxe9j6m2kYdaqQO37hkAyfNM5eqKv7vgedroVVGUOo NF6pKgTECpKKkg5bEiVfASYQzRJBbbDFr/Tg++9M7CVkjleEs/UJ5J/9eBeAG/JWUNLe T0kWjz/0+gz3EyOaQ49jUk9YE0ycA+XCMD0S2r9aCxNee6epjlwZGr81jfSWjoYU2kBQ iHsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821264; x=1701426064; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zO78n1C+IUUd4qQDduar+mDDdO70FT4CA4aVaSqaf3o=; b=UnEygCut4OdrZBppxtS6JXLHQLZZxqdN/ugwtizy0dAQCMvF8HL7wB90U+Fj7AeVW4 SiV234VhMBh6Qeupz2rXdayJObYxQn1UpotqVy660xqEZ05XOv2jtElBH7wYDjg5i6wr tafEkhzAJtR2LDMm13MAViJd58pmpyGOF5crykZs1lry5vC4B8oYRM2xY3jzvrIgLDQI 9V2Fs3W6O5dPBXilhs/Sl7FE2s800hu4p2Yi1yRjdc+p3khA7s7ZJBVXFhG3MvIqMW9w 8DmnUnTwR40/SJx4pndq/0UkleSbhnCauKVGeygFLryC3L4fbAtt61XoWdzJpELhPzKg 3tCA== X-Gm-Message-State: AOJu0YyS3cehsipXHc6yYZBCNFnq7Z6sUF7y2DTQoC4d1iivL84SokLM dj2aDTrEbArr1FvNRMOuEotpltKXMbrHylAw//iK1Ej1jryWb8uxD8uCmmWx/f1/KkucUKESwY6 gx5jge7EJTc+OBZ/wBu+5CSYyq6mbpU4N+yBgWyXROJVBAEBE4th3luDRp6gD87YKzOxHNFI0lI A= X-Google-Smtp-Source: AGHT+IFds6lDNxC+wfxNjjQ4BJ0IOb3Lhz4+mu1JWEbRpip6yVBtW9wTPhlgHd/mpJ5KZ+rN2yoADRiF X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:458d:b0:409:687f:3419 with SMTP id r13-20020a05600c458d00b00409687f3419mr45473wmo.0.1700821263775; Fri, 24 Nov 2023 02:21:03 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:02 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3393; i=ardb@kernel.org; h=from:subject; bh=nYRIcwIYlsV304zLeGlXyA6jIf75Zhb+sT/wec3N1GQ=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYj4/24Ypr9ZJbud4t7vd+0aRXNH+M5+PfskzELu9V WDK58XRHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAi55YxMjzom9167kOz5uQd vy8WeNpt+/zyaEh9S/iNW2kGd3lXWM9j+B+yvtnuePqOsj5Dh4/XFvp9SQ45cIf1/JoHR1pWn2h I2soGAA== X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-63-ardb@google.com> Subject: [PATCH v5 22/39] arm64: head: Clear BSS and the kernel page tables in one go From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022106_258480_6B81137D X-CRM114-Status: GOOD ( 15.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We will move the CPU feature overrides into BSS in a subsequent patch, and this requires that BSS is zeroed before the feature override detection code runs. So let's map BSS read-write in the ID map, and zero it via this mapping. Since the kernel page tables are right next to it, and also zeroed via the ID map, let's drop the separate clear_page_tables() function, and just zero everything in one go. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 33 +++++++------------- arch/arm64/kernel/vmlinux.lds.S | 3 ++ 2 files changed, 14 insertions(+), 22 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index ca5e5fbefcd3..2af518161f3a 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -177,17 +177,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) ret SYM_CODE_END(preserve_boot_args) -SYM_FUNC_START_LOCAL(clear_page_tables) - /* - * Clear the init page tables. - */ - adrp x0, init_pg_dir - adrp x1, init_pg_end - sub x2, x1, x0 - mov x1, xzr - b __pi_memset // tail call -SYM_FUNC_END(clear_page_tables) - /* * Macro to populate page table entries, these entries can be pointers to the next level * or last level entries pointing to physical memory. @@ -386,9 +375,9 @@ SYM_FUNC_START_LOCAL(create_idmap) map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT - /* Remap the kernel page tables r/w in the ID map */ + /* Remap BSS and the kernel page tables r/w in the ID map */ adrp x1, _text - adrp x2, init_pg_dir + adrp x2, __bss_start adrp x3, _end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 mov_q x5, SWAPPER_RW_MMUFLAGS @@ -489,14 +478,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) mov x0, x20 bl set_cpu_boot_mode_flag - // Clear BSS - adr_l x0, __bss_start - mov x1, xzr - adr_l x2, __bss_stop - sub x2, x2, x0 - bl __pi_memset - dsb ishst // Make zero page visible to PTW - #if VA_BITS > 48 adr_l x8, vabits_actual // Set this early so KASAN early init str x25, [x8] // ... observes the correct value @@ -782,6 +763,15 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu + + // Clear BSS + adrp x0, __bss_start + mov x1, xzr + adrp x2, init_pg_end + sub x2, x2, x0 + bl __pi_memset + dsb ishst // Make zero page visible to PTW + #ifdef CONFIG_RELOCATABLE adrp x23, KERNEL_START and x23, x23, MIN_KIMG_ALIGN - 1 @@ -796,7 +786,6 @@ SYM_FUNC_START_LOCAL(__primary_switch) orr x23, x23, x0 // record kernel offset #endif #endif - bl clear_page_tables bl create_kernel_mapping adrp x1, init_pg_dir diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 8dd5dda66f7c..8a3c6aacc355 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -311,12 +311,15 @@ SECTIONS __pecoff_data_rawsize = ABSOLUTE(. - __initdata_begin); _edata = .; + /* start of zero-init region */ BSS_SECTION(SBSS_ALIGN, 0, 0) . = ALIGN(PAGE_SIZE); init_pg_dir = .; . += INIT_DIR_SIZE; init_pg_end = .; + /* end of zero-init region */ + #ifdef CONFIG_RELOCATABLE . += SZ_4K; /* stack for the early relocation code */ early_init_stack = .; From patchwork Fri Nov 24 10:19:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467571 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DE18CC636F3 for ; Fri, 24 Nov 2023 11:31:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=FqeFFlC6RZ6kJu5gThRkfBMER62/m8OUp6W01Fj18+8=; b=kZU5ntvcBiOTVmWdEPkTjE8U2f qeJY9udlZVPfMZ8I+dpfwSf2ypvDAtakeYPKSWUi7GXc8rRYxdjTQk4+Cb1uYpBkQW8u+FwyP/7oe Wv2tuScr5OwjXY7jUg+ykhuAQaXkNWukJ94NJ9+LjPBHzFrIf6uU5soWKUQxLuNtJYoAxL7NZDux9 Y8uc7d0ehzRPsIFK0lkcAqJ1NDwclMRt71a59ejm3wBMDillV/RfI5aoyS3TKiCYPK3vwEWvSKwfL OjL+OeNiyfkA0kQqnppMds1F6fdpNs+NNUyZnFAzhEUPtQl9sD8bbfyuykIPiyTisZeIxnqW5I0yG Ibw57jYg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6UPY-006yVD-08; Fri, 24 Nov 2023 11:31:52 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJF-006p1l-0E for linux-arm-kernel@bombadil.infradead.org; Fri, 24 Nov 2023 10:21:17 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=0vYG5XgKCknhzw1oKvHG5TuM/4N/BkCC3lnuRkCd8+8=; b=HROfDOAf+bh5XV6rrjUEfuqguq r3Xzp2CY9dqEjRo6xTSifuTFxcmUY79Eu2l2VSPYtDC9nDjS6+Givwfq91FmqGC0kflQG90AhY1Yu mvAF1MxAEhM5PAeKz4XZOwCIW8svBieHMieR/r1/OhcI8jVGNaiz6rgcsM0dzlgBsFOibMfgpAVTx 5URWyqlqccS361nFWY5VIZZfJF5ar56gtekdkASibQWI0BZMB8lNBLbU5e/gFeDKIp6pS9bN7zZ4s QEslZvjgshLlamDpAORnfv1xPzeBNUUdqvdt6kdT4ewUQAzsOcSNacV0IkljCZGJ6zyMH2Z+D0Ggn L3Ef0b9Q==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJ9-00DqTM-1y for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:14 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-da03c5ae220so2105528276.1 for ; Fri, 24 Nov 2023 02:21:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821266; x=1701426066; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0vYG5XgKCknhzw1oKvHG5TuM/4N/BkCC3lnuRkCd8+8=; b=Atn9aO2ATfNCjxkQuFFvzzEe4J/2iZ0v32kLl9ZI6V7AhGJ1J1TF1qSGxmFHjG+F1m 9hdZ+Tzi5WBi6wNL8WjJC3jWUbX/l9XkjD6kVM05R0HNPPBLwpsjNhV1aR5SIE7EHn89 4qGBSStTlG/ea8BVhRp0+iocPYDiF7bRPZ/Pc1ITrdbWRUjCSg1xJyZ2i/6Upw/VeBFR BizAPR6AhRXc9/ZonrWxfQkITvqviHfIWdhQQVMXIa1BT0kWOPVHHOoCDbGu94TC+W8p UOs0KDrTMeuUaeUr1HSs1RP1zS3MZoqVAcSa486X3dBhlAE+VmAuEsZ4/p3ZDUrgRzeV eSHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821266; x=1701426066; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0vYG5XgKCknhzw1oKvHG5TuM/4N/BkCC3lnuRkCd8+8=; b=mXf2zRg398JcrUQ/riH6RfV3/rke3NRmFVL20wpjrBzVVYqKI46rV5mbiTYJJHaP9y CUQSy+LntWLHMPSQauu1DYA5br5/lhIjt998oi3REzl6VxUI89Py9pCuSSx1T0bRI3Yq 4uKFlQ+PIj9zLeTkFQ6X9ffw3G1A9oXNbG00CA4+F18DTjCBurnmbIosir1HHmo/QaxI rSGzHu5qADarBx41rB4ijSlWtQOoLoCiTEIx0FbMLkr+Bip8//CeXZKm175zZtg7Us3e CTV7fjE/lf4XJJQbsr/z8SQH/CvvNJWAg2ZUIbeR1fRCJPvAY0z7s/ALkuo4CBnJa7LW VqJQ== X-Gm-Message-State: AOJu0YwbNtj1C60Ox1ApLIjFt+oXpwtDxsvNS4H95APdMkETemtYRqsM gtf2WHJ3tvd47QfMLPlddwWvA1hS9HbGA4kFN3oPJtUbWWxz4ALXUZvyvCSc1aeyLpmRA7n4mH6 hJx56N+MUodBtzaU/pRd0A4n1fAL40Rk0hPFx63gHdN622bZ0erkigSbbwzsU3WYqVHmtjfXmP9 M= X-Google-Smtp-Source: AGHT+IFbez7SHri59dtj5nr6p8vKPW0dnuPccdV+0legLXsI9OaMBOr3+3F+Jmvy9r25g+vkh/jhWOPu X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:9b82:0:b0:d9a:ca20:1911 with SMTP id v2-20020a259b82000000b00d9aca201911mr218175ybo.4.1700821266396; Fri, 24 Nov 2023 02:21:06 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:03 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1681; i=ardb@kernel.org; h=from:subject; bh=qIkPpxR3MHu+G49P+6aS8D50R5WFMTo+okAIbuN9jSw=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYuH3lBz5f3pvxFjW3dijtyn9oePG5C3r7vaLV2XX3 6lJEl/WUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACaiLsbwTyPa/em1S2mTHq9+ aHDRV2KPm+63e+xbH68oLjCUbpvYcpqRYXpXftwjzZ6EH1lKeUf/BfdaL8nY9Gnf1pt1Ky1le7d cZwMA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-64-ardb@google.com> Subject: [PATCH v5 23/39] arm64: Move feature overrides into the BSS section From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_102112_192346_1E8399B1 X-CRM114-Status: GOOD ( 10.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In order to allow the CPU feature override detection code to run even earlier, move the feature override global variables into BSS, which is the only part of the static kernel image that is mapped read-write in the initial ID map. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/cpufeature.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 646591c67e7a..b95374387946 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -655,13 +655,13 @@ static const struct arm64_ftr_bits ftr_raz[] = { #define ARM64_FTR_REG(id, table) \ __ARM64_FTR_REG_OVERRIDE(#id, id, table, &no_override) -struct arm64_ftr_override __ro_after_init id_aa64mmfr1_override; -struct arm64_ftr_override __ro_after_init id_aa64pfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64pfr1_override; -struct arm64_ftr_override __ro_after_init id_aa64zfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64smfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64isar1_override; -struct arm64_ftr_override __ro_after_init id_aa64isar2_override; +struct arm64_ftr_override id_aa64mmfr1_override; +struct arm64_ftr_override id_aa64pfr0_override; +struct arm64_ftr_override id_aa64pfr1_override; +struct arm64_ftr_override id_aa64zfr0_override; +struct arm64_ftr_override id_aa64smfr0_override; +struct arm64_ftr_override id_aa64isar1_override; +struct arm64_ftr_override id_aa64isar2_override; struct arm64_ftr_override arm64_sw_feature_override; From patchwork Fri Nov 24 10:19:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B92B1C61DF7 for ; Fri, 24 Nov 2023 10:22:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=TC7oWd0xzWK7MFQwMKg4l/N2mBY84Z3y9DZXreE3T2M=; b=C/VWhDwxqaaRswPS1NsoZJFfwI W0BIkj2b8j/ZJCnrnAnZmuJmrwxJzElQ8OmGkdGyQmZKjjFSeH6zWQ0yg5c5xcQta94NPXlR+5Luf OjVUmQjI/PJ2vGZcfTgZAThRjPW4LUM61H9qgDi5RYFej8NXtWScMcUnW5FYJoAMk0TVAEcJG1vq7 aplWTB1zOfH9ozPygoK5puoqdYUZCe2GGdfe+QAisx0sCXR1upVtCmzaHJvZlFG2/US+VOm0woO9o RsQanXHuRANotdlCShu0ObAHSaRbt/XkTdcsX7QuaJwdIs10cBtQbMFJOJqnANs2pdsPfXXAfU0XK ykH92pjg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJX-006pGB-2o; Fri, 24 Nov 2023 10:21:35 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJF-006p1t-0E for linux-arm-kernel@bombadil.infradead.org; Fri, 24 Nov 2023 10:21:17 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=cngrmd8oycagSPoZSvoHKPsQhFg1xaF7tU7fqPfLIP0=; b=ZUe01ABFK3ksdxSLYTAtSqAnlP F8JoVj4GhpMIoa+vZb9iEYujjwrsJZVCNNunweCBA2BdX9ZE/W2X3E7MDa8FlXQ9l2O3fb1u+8RzC zCpwysUVsnF499IiiXRkfqzxgzvdC8W8jy0qGCnOZym5A8YNb6yvuGKdjl1vMd2+qKibEVuaMYZPt ijL/U713yB1kNKw5g6Z9nLInbTjsCDQXzV6KY6Ndk4agv7oyWPjpaOMnXM27xebpxjpKY2Fuufkhy YSGZp8pakFfS3fkHPEjtqYu/LLx6l5dw4yB72xBEJ+94lTv5mp55p/90OBZeyz0f+WKf7FsZEgH/L GKycU7Iw==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJ9-00DqTQ-2B for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:15 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-da31ec03186so1221376276.1 for ; Fri, 24 Nov 2023 02:21:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821268; x=1701426068; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=cngrmd8oycagSPoZSvoHKPsQhFg1xaF7tU7fqPfLIP0=; b=oRr7vcfwWlsfal0fymnWVHJkxujbvXdlqDT2WV0l8DAAr7+CjONCAsMcR0BWW/1H5v 6BO8I90fnAzk/71s+E7pehL2gV9u/Xcs52q7qY4Mc28NNx3MeJFc39ZP3nl/1kXpff+M BQj9HcX9vyDvzIZHSrfQEmMqc59nXeiSrSOlAiOAmhij0KiiXUymKTozy40OaYDKv6Zr AD3ZTcE+Gnl/T5DeJ3h+YZBDPobMgWLpwjR7K93V1Nr3KxJHsYfRiZhKocwTm1t8b5jm AUj01K7TNzOzPGSh1jMxaBUDZ3vfAoR/DvqB2WD4xzxjnCnQwyAW5CoD0z91qiDhxV00 nKpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821268; x=1701426068; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=cngrmd8oycagSPoZSvoHKPsQhFg1xaF7tU7fqPfLIP0=; b=lQDJGh2iV3E/XPKs1VxCL05abYSA5RtRbI0Pgt3ugnb03H5M1NkVE80eGwFIhK5fhP qghuRZO7eGPSVYvamX+GQYlQxteD9V/xvrYepZQN0Dj8Yef/VLLBlyjhYRdaFZV7+PGk /q2gIs8cbp6Y/St1lEENr9jTv1glsc3RO2E+kTrmNXvCJZK6GSmr47e16iaCagY6HeHk 4yoQEUxo7JwgeHF9pY8q1Xu238zM/I8dwWYFjFoFVbClxCeM3x+mdokVlgt0vc2hXXZg ipTBrL+Cl7Qyd/d13z3H/Z849ZpsmDP3AirzRXBe2jwXnmNfpJbc8O83lwntd8UfrbpB ZFwA== X-Gm-Message-State: AOJu0Ywkhb88jHXWR2SgB2vllDBD6c9MtShjMZiB58QfkAyu60nuiABE mIP9SUox56NeLnMyxz0ioZYV+/8AMvINLb2VH3prBL9vQhLDiuytdyEBXrRdfLzaZZSMqvZSPqj oPyb0tlwJP+xPrXT12Cg695vFndeT8dkP/8rZ8XfuDvxcdSsJOw2sE1Pbc0/WSl3us8pOLG+YnP g= X-Google-Smtp-Source: AGHT+IF+Tzfknw68/7awEs3CcmlIetn8vgw9BLHRYQYb+aZSnRl+IbRdYQ53JzgYw1wGMf7q4bGBcyBo X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:dd07:0:b0:daf:3e84:d016 with SMTP id u7-20020a25dd07000000b00daf3e84d016mr78242ybg.2.1700821268525; Fri, 24 Nov 2023 02:21:08 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:04 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2829; i=ardb@kernel.org; h=from:subject; bh=AOSTHt9JvO/4iRzuZocBqHtffwUhdcZrrydQAmtcn8o=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYrHe/cUx//sqdnJcZrVQmpjgumCNa/CKvkN1IX+3H tRc9baro5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAExkvj/D/5yMruuFLkWXH2p8 3b53/eXDd29OmGYbr3vT4ZOIjuaZi6KMDDsk78Sxy6xhfDvnwfzVKk9E48/vfsx4N8G4Y0/WQwH xMBYA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-65-ardb@google.com> Subject: [PATCH v5 24/39] arm64: head: Run feature override detection before mapping the kernel From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_102112_043273_565D48D2 X-CRM114-Status: GOOD ( 13.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel To permit the feature overrides to be taken into account before the KASLR init code runs and the kernel mapping is created, move the detection code to an earlier stage in the boot. In a subsequent patch, this will be taken advantage of by merging the preliminary and permanent mappings of the kernel text and data into a single one that gets created and relocated before start_kernel() is called. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 17 +++++++++-------- arch/arm64/kernel/vmlinux.lds.S | 4 +--- 2 files changed, 10 insertions(+), 11 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 2af518161f3a..865ecc1f8255 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -375,9 +375,9 @@ SYM_FUNC_START_LOCAL(create_idmap) map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT - /* Remap BSS and the kernel page tables r/w in the ID map */ + /* Remap [.init].data, BSS and the kernel page tables r/w in the ID map */ adrp x1, _text - adrp x2, __bss_start + adrp x2, __initdata_begin adrp x3, _end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 mov_q x5, SWAPPER_RW_MMUFLAGS @@ -491,9 +491,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif - mov x0, x20 // pass the full boot status - mov x1, x22 // pass the low FDT mapping - bl __pi_init_feature_override // Parse cpu feature overrides #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS bl scs_patch_vmlinux #endif @@ -772,12 +769,16 @@ SYM_FUNC_START_LOCAL(__primary_switch) bl __pi_memset dsb ishst // Make zero page visible to PTW -#ifdef CONFIG_RELOCATABLE - adrp x23, KERNEL_START - and x23, x23, MIN_KIMG_ALIGN - 1 adrp x1, early_init_stack mov sp, x1 mov x29, xzr + mov x0, x20 // pass the full boot status + mov x1, x22 // pass the low FDT mapping + bl __pi_init_feature_override // Parse cpu feature overrides + +#ifdef CONFIG_RELOCATABLE + adrp x23, KERNEL_START + and x23, x23, MIN_KIMG_ALIGN - 1 #ifdef CONFIG_RANDOMIZE_BASE mov x0, x22 bl __pi_kaslr_early_init diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 8a3c6aacc355..3afb4223a5e8 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -320,10 +320,8 @@ SECTIONS init_pg_end = .; /* end of zero-init region */ -#ifdef CONFIG_RELOCATABLE - . += SZ_4K; /* stack for the early relocation code */ + . += SZ_4K; /* stack for the early C runtime */ early_init_stack = .; -#endif . = ALIGN(SEGMENT_ALIGN); __pecoff_data_size = ABSOLUTE(. - __initdata_begin); From patchwork Fri Nov 24 10:19:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467574 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A0C6EC61DF4 for ; Fri, 24 Nov 2023 11:31:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=wDqd/gGDvMGtMO1/KD7n1W8okVzZ73kGrZ0PzlAcs8Q=; b=lhy0mHhaYNDlFeK1XoR/a0YFTd jvs40MKElZZ3wp7zUP3JJmYjadh9LMWSz0jS9c5hJwizs/Js574y+kzNCQ3S9eBJF7rlAp00qCzes xcADMeAsMf1QunWNgWOzmExtZQHRkTOvqS6kUQfL2f30Vaky/jgBy3bOxqWOajlbBb7SVFQF7cUXZ SGkenoE8sQtL5gOi50jwDkW3vGZBBhyhGR8ug7yz4nKOhuZoNC4aDOYU2R0pXPfX2XZoOiTkPAum0 KPOYKyK77x8PUOSqKzMqiUnYpJsYFhzK2xqp+fNJGJUZWmOSp3843ETNNzTSMB1iKwBRjg2bYAjpE T8je/m8Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6UPa-006yY0-1O; Fri, 24 Nov 2023 11:31:54 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJA-006oyU-2T for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:26 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5ae5b12227fso22942637b3.0 for ; Fri, 24 Nov 2023 02:21:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821271; x=1701426071; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=oTb5YT02RwqmCRs+XMdOFbLPPfCLhuRkcodO2emiEKQ=; b=o6Yv2Ggi3pJQrrqFJo8tK8/jQXxtTFym4bIQ4XZUjZ1g38GuZlrZNp1TYWmQwceVYA jtKfvzLTmqlJk/sSQaN7xyVUTPtn8H1ABBJbmynekoYTpIrGR/QtlOdjRfFbCUAOQNyf z6tzDToGuIX8F481djhmUYEDHNetuXHN6MDcck4q4pVzZwhDbn+ir12bHlAIor7BstpB Q6HpqKMv8OFvt2TEw7T2FvkNC23/gZ7v77CO19/Dq2YevHMubXkZvOGKy6LFEu71yC5V fJdi8JI0u1qxWSTU8srBIHrGDWSDNNkfmhhzYq4gAqEshNQX/mNzploXuwJ+bJH4Ue9q IdtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821271; x=1701426071; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=oTb5YT02RwqmCRs+XMdOFbLPPfCLhuRkcodO2emiEKQ=; b=Ggoqse3o8TXCkPsKUZ7bhj4AE/0ocxgGoNXp9YV/cNw5VEhjDZWiNzPEWVJmxc3rRT gEBti8GuGPNsMFR5NiguNuIVdcLwfnTH8MKzFUdAsRfDlCMhkSyqmO9j/xBQF+K091Mg 02ler9Vn1k4aL1Bv0wAsQiuU/07AKlStnW2fZPH4X1R/UnOUF50oxcbS6qVYyFEK2DwU B/8BZVtoKGQ9jc/4YF/h3eDjUaivCr7oZEUr5azXj2CuMdeCB7snvsi5l1GZJo6VcitG l+geEYjrRBBvbYX1D8kBubmFk0Nw5xKgZ8v/ZmCSS/cFtuPOFN2OBrIJ1UkICC4pTCD2 JTVg== X-Gm-Message-State: AOJu0YwVvg3K97P+vtXM1kmrPs4UpO1OGtjn8iey/nPo8fWxC0SRppQ1 qnk3u7FGXIYQ3ZfSSA9cbhMO8ChyFztG5tLPXhEmPns+VAFLDZlggpcAK64xA/laLEVzzxe0lh9 VK08Y1Cfn42TDYYhFD/kJ/RMqXrLRFTcNmValfWZMOtL3aHuDKLRcMn3MHyTGxTp3uqJV+6JN9D w= X-Google-Smtp-Source: AGHT+IEoLpNDf0BTqWnPT+rYGd7qhqdp5E2zt1lZFNdHKH7/lL+JBG/dk2nMXQUTWWTPTEGBI9hz9QhN X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:690c:3603:b0:5ca:e7e5:e8b9 with SMTP id ft3-20020a05690c360300b005cae7e5e8b9mr79755ywb.9.1700821270986; Fri, 24 Nov 2023 02:21:10 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:05 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6307; i=ardb@kernel.org; h=from:subject; bh=+CAandW2REwobvpWTcepqeb8aChD0Vxr0cE2KhPU7EE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYons496u55PFf7k8tMx51LO26k/wDOa8/WcEVSbl8 O7XvirUUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACYiw8fIMOsIQ8vynw35xkwX /9jlvb3zu/jLj7ypFYyt4btufd/YJMTI8HwaQ1fGzqPJ1TMeqPlNOXvJQEfgbfLKqj7b52nrUid +5gEA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-66-ardb@google.com> Subject: [PATCH v5 25/39] arm64: head: move dynamic shadow call stack patching into early C runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022112_851481_C827DD58 X-CRM114-Status: GOOD ( 18.09 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Once we update the early kernel mapping code to only map the kernel once with the right permissions, we can no longer perform code patching via this mapping. So move this code to an earlier stage of the boot, right after applying the relocations. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/scs.h | 4 +-- arch/arm64/kernel/Makefile | 2 -- arch/arm64/kernel/head.S | 8 +++--- arch/arm64/kernel/module.c | 2 +- arch/arm64/kernel/pi/Makefile | 10 +++++--- arch/arm64/kernel/{ => pi}/patch-scs.c | 26 ++++++++++---------- 6 files changed, 27 insertions(+), 25 deletions(-) diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index 3fdae5fe3142..eca2ba5a6276 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -72,8 +72,8 @@ static inline void dynamic_scs_init(void) static inline void dynamic_scs_init(void) {} #endif -int scs_patch(const u8 eh_frame[], int size); -asmlinkage void scs_patch_vmlinux(void); +int __pi_scs_patch(const u8 eh_frame[], int size); +asmlinkage void __pi_scs_patch_vmlinux(void); #endif /* __ASSEMBLY __ */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index dc85dc2ee4ed..14b4a179bad3 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -71,8 +71,6 @@ obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o -obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o -CFLAGS_patch-scs.o += -mbranch-protection=none # Force dependency (vdso*-wrap.S includes vdso.so through incbin) $(obj)/vdso-wrap.o: $(obj)/vdso/vdso.so diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 865ecc1f8255..b320702032a7 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -490,9 +490,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init -#endif -#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS - bl scs_patch_vmlinux #endif mov x0, x20 bl finalise_el2 // Prefer VHE if possible @@ -794,6 +791,11 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RELOCATABLE mov x0, x23 bl __pi_relocate_kernel +#endif +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS + ldr x0, =__eh_frame_start + ldr x1, =__eh_frame_end + bl __pi_scs_patch_vmlinux #endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index dd851297596e..47e0be610bb6 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -595,7 +595,7 @@ int module_finalize(const Elf_Ehdr *hdr, if (scs_is_dynamic()) { s = find_section(hdr, sechdrs, ".init.eh_frame"); if (s) - scs_patch((void *)s->sh_addr, s->sh_size); + __pi_scs_patch((void *)s->sh_addr, s->sh_size); } return module_init_ftrace_plt(hdr, sechdrs, me); diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 7f6dfce893c3..a8b302245f15 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,7 +38,9 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := idreg-override.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o -obj-$(CONFIG_RELOCATABLE) += relocate.pi.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o -extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) +obj-y := idreg-override.pi.o \ + lib-fdt.pi.o lib-fdt_ro.pi.o +obj-$(CONFIG_RELOCATABLE) += relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o +obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/patch-scs.c b/arch/arm64/kernel/pi/patch-scs.c similarity index 91% rename from arch/arm64/kernel/patch-scs.c rename to arch/arm64/kernel/pi/patch-scs.c index a1fe4b4ff591..c65ef40d1e6b 100644 --- a/arch/arm64/kernel/patch-scs.c +++ b/arch/arm64/kernel/pi/patch-scs.c @@ -4,14 +4,11 @@ * Author: Ard Biesheuvel */ -#include #include #include #include -#include #include -#include #include // @@ -81,7 +78,11 @@ static void __always_inline scs_patch_loc(u64 loc) */ return; } - dcache_clean_pou(loc, loc + sizeof(u32)); + if (IS_ENABLED(CONFIG_ARM64_WORKAROUND_CLEAN_CACHE)) + asm("dc civac, %0" :: "r"(loc)); + else + asm(ALTERNATIVE("dc cvau, %0", "nop", ARM64_HAS_CACHE_IDC) + :: "r"(loc)); } /* @@ -128,10 +129,10 @@ struct eh_frame { }; }; -static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, - bool fde_has_augmentation_data, - int code_alignment_factor, - bool dry_run) +static int scs_handle_fde_frame(const struct eh_frame *frame, + bool fde_has_augmentation_data, + int code_alignment_factor, + bool dry_run) { int size = frame->size - offsetof(struct eh_frame, opcodes) + 4; u64 loc = (u64)offset_to_ptr(&frame->initial_loc); @@ -198,14 +199,13 @@ static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, break; default: - pr_err("unhandled opcode: %02x in FDE frame %lx\n", opcode[-1], (uintptr_t)frame); return -ENOEXEC; } } return 0; } -int noinstr scs_patch(const u8 eh_frame[], int size) +int scs_patch(const u8 eh_frame[], int size) { const u8 *p = eh_frame; @@ -251,12 +251,12 @@ int noinstr scs_patch(const u8 eh_frame[], int size) return 0; } -asmlinkage void __init scs_patch_vmlinux(void) +asmlinkage void __init scs_patch_vmlinux(const u8 start[], const u8 end[]) { if (!should_patch_pac_into_scs()) return; - WARN_ON(scs_patch(__eh_frame_start, __eh_frame_end - __eh_frame_start)); - icache_inval_all_pou(); + scs_patch(start, end - start); + asm("ic ialluis"); isb(); } From patchwork Fri Nov 24 10:19:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467575 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7ACCFC61DF7 for ; Fri, 24 Nov 2023 11:32:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=8NYOu5nPCLoWpVHFE0hMg1SvuKInGDNI5YRJBAmlkJI=; b=P5PLg3VnECm+iFMz58selk7uoE 9EdEST90OE5aXroAmYRqqgg2H0codLA5sX2Gu4AXU3ycPvdAJ+6C2+8/5ZIJlE8Bb9j3pFThOQMYA Re1ZwJxhhBKU4/dYPiCQ9nuMEUr1F9d41efu/na9WAAQ5Ri0f7kR9/0b7oRtQxmniL0d9I4eXZAhC zjStqBbIx4XLfs9sHIp5XbUG0JY1C4xU9oNNHm4UGbedEM0Oyv7QI92vCIy5f6uZHApmVxdbmMABp wVojkwvF3E/s47s4FxOJ3TJ5ahI3Ksc3iEcDFANqHSk8Y3gxj29Z+zX0Xj6IfTUopxHghwHWQdioX TuO2xJVw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6UPb-006yYy-0g; Fri, 24 Nov 2023 11:31:55 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJD-006p0R-0p for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:28 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-db40f549659so1961019276.0 for ; Fri, 24 Nov 2023 02:21:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821274; x=1701426074; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=S3e5I+bgR/tzZLdbHycPVzYs/+XiaKWgnJ1R1oh4B0o=; b=Kgywlau0SEx6xAFl1i4/8QIrUelKnEdKMEy3U/7NfUwRsTtmZ2lw6kW2+fadK6f5LO OT2WbZVF31pMGSBudDRPWOrkvltH6ozvI85e5+V+IsNPKibVB7FbjtmJqmKkWglSS2t/ LghxZBTE+j5FLvcRmhbuMZtAQaI1Q5+Erb7gRmC/gRzNCcZm2MDVGrp1C2BLE0P9nO++ deikn0bEq3WiCVhUgxH5eGDVnbIsxRgj4LSRXMqPo0oUzVdv/Frwel0KMwI/qG6tWH3Q 8OmawPX4j/F+lSnEasPxwVKETRuSgc1TWL+duu6S4LETyKN4piQhbEAXFO9KQIdWBgxO heVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821274; x=1701426074; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=S3e5I+bgR/tzZLdbHycPVzYs/+XiaKWgnJ1R1oh4B0o=; b=dEsdCJnXISLM7D2+HUb8ytauuiNGld2Q7P3lgFQnWRfVJ22mF3ZlQsmwtkM1I4aeat +SV7JAYsw245Pk7ajNu7GaTXLRf+N89bFF9b1j255rDabFWySbwXo90+HopYIibiHbA7 P96ugX0glEz6Cbr2mWjvMIRSJfie2qgiJIEQK2wLGHkEzsVYjGwoVg//anUjhEn3kgeh fLAA0I5sPRcAXU6D7KQN+n6Fk71/jme/rx/s5kBncBn4lM6gg6eaarW5Q7ekctcCV5Zj o8gCx04S9DzLFQ5+JcMb4dvpZr8IvoBN3WyIQ8tSdPdSOfNOKcmL8TxKQ0xZRE7QzETe TPww== X-Gm-Message-State: AOJu0YwBifO3bw7T+b05oLOF2f16rlB7ZeZkAVgXER2cppoQunMnv/fP f+t0dFMpJnX2CeKT02mFHB9DK6Kqvut/yRLCkKodI7Sh4gKcQ6rzQBoEYNyo9FSjx3ZYwrzR7wM XDa2Ohi+u037X3DrXC2eBgHqZ0yFG1u1P5pOQ6gG/NVPVi1vcfRFQobz7deA7EGkMQYqj6Yx+Qa c= X-Google-Smtp-Source: AGHT+IG6w+jGBjeF64U075K8IqGtdu27ZhUbrQU7jEWCoZ5EI/S67YfAY/HBOfqNW4wwp+ZTkVmNXf3I X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:cf8e:0:b0:daf:34be:1e83 with SMTP id f136-20020a25cf8e000000b00daf34be1e83mr70336ybg.2.1700821273301; Fri, 24 Nov 2023 02:21:13 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:06 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3526; i=ardb@kernel.org; h=from:subject; bh=lBhgC2RIE3W64cp9ghnBvS79VDuJ3eFWjD45q0X0UGw=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYqnZh/cWD8/LSajymIfXrnB/I3DUo9BH4mO9TG+Eh sSujOkdpSwMYhwMsmKKLAKz/77beXqiVK3zLFmYOaxMIEMYuDgFYCLXkhgZtl+7YfB1/b3d2w7G 8qxse/at57XR03k//2+/+O7p+oszGF4zMnyQO9D0em/F0yk7/G4pntrCsaFJQDlvnZKyxxEXtbI NojwA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-67-ardb@google.com> Subject: [PATCH v5 26/39] arm64: kaslr: Use feature override instead of parsing the cmdline again From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022115_363258_207EF657 X-CRM114-Status: GOOD ( 16.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The early kaslr code open codes the detection of 'nokaslr' on the kernel command line, and this is no longer necessary now that the feature detection code, which also looks for the same string, executes before this code. Note that the pseudo-feature's mask can be disregarded: it is used for true CPU features to mask the CPU feature register, not the value of the override. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 8 +++ arch/arm64/kernel/kaslr.c | 4 +- arch/arm64/kernel/pi/kaslr_early.c | 53 +------------------- 3 files changed, 10 insertions(+), 55 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index f6d416fe49b0..77ed3b28cbc6 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -909,6 +909,14 @@ extern struct arm64_ftr_override id_aa64isar2_override; extern struct arm64_ftr_override arm64_sw_feature_override; +static inline bool kaslr_disabled_cmdline(void) +{ + if (cpuid_feature_extract_unsigned_field(arm64_sw_feature_override.val, + ARM64_SW_FEATURE_OVERRIDE_NOKASLR)) + return true; + return false; +} + u32 get_kvm_ipa_limit(void); void dump_cpu_features(void); diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index 94a269cd1f07..efbeb8356769 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -16,9 +16,7 @@ bool __ro_after_init __kaslr_is_enabled = false; void __init kaslr_init(void) { - if (cpuid_feature_extract_unsigned_field(arm64_sw_feature_override.val & - arm64_sw_feature_override.mask, - ARM64_SW_FEATURE_OVERRIDE_NOKASLR)) { + if (kaslr_disabled_cmdline()) { pr_info("KASLR disabled on command line\n"); return; } diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index 167081b30a15..f2305e276ec3 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -16,57 +16,6 @@ #include #include -/* taken from lib/string.c */ -static char *__init __strstr(const char *s1, const char *s2) -{ - size_t l1, l2; - - l2 = strlen(s2); - if (!l2) - return (char *)s1; - l1 = strlen(s1); - while (l1 >= l2) { - l1--; - if (!memcmp(s1, s2, l2)) - return (char *)s1; - s1++; - } - return NULL; -} -static bool __init cmdline_contains_nokaslr(const u8 *cmdline) -{ - const u8 *str; - - str = __strstr(cmdline, "nokaslr"); - return str == cmdline || (str > cmdline && *(str - 1) == ' '); -} - -static bool __init is_kaslr_disabled_cmdline(void *fdt) -{ - if (!IS_ENABLED(CONFIG_CMDLINE_FORCE)) { - int node; - const u8 *prop; - - node = fdt_path_offset(fdt, "/chosen"); - if (node < 0) - goto out; - - prop = fdt_getprop(fdt, node, "bootargs", NULL); - if (!prop) - goto out; - - if (cmdline_contains_nokaslr(prop)) - return true; - - if (IS_ENABLED(CONFIG_CMDLINE_EXTEND)) - goto out; - - return false; - } -out: - return cmdline_contains_nokaslr(CONFIG_CMDLINE); -} - static u64 __init get_kaslr_seed(void *fdt) { static char const chosen_str[] __initconst = "chosen"; @@ -92,7 +41,7 @@ asmlinkage u64 __init kaslr_early_init(void *fdt) { u64 seed, range; - if (is_kaslr_disabled_cmdline(fdt)) + if (kaslr_disabled_cmdline()) return 0; seed = get_kaslr_seed(fdt); From patchwork Fri Nov 24 10:19:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 69351C61DF7 for ; Fri, 24 Nov 2023 11:31:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=a6qDLaSJUZv351sLoognBkjakXwP5p3wZRuZ4ieBgTg=; b=jtUuHEu/WGTp6z+f1S5Zwkc4fS yYm64VL9eZJ2kQHzCtwxOyZQRJogJ5ITdAy/8iP3puhfzbi3Ka41vKVfF0QIVvLAkq7RusY3oF7FG V2+y0HL6WYpteJhqWKWNmj2SP3a3A0+RIYdRZTGs0ysgYQ72MJUUPPGwCurz3lqh6YanySJjkg4NY R3gGNY2afN4fiskxSt7d59B9KqDbDesSLlz3xd9jI2Yru4mFI/htpGldrWM7c/7k9VM03gU9xWRGz cRwReNMPxBcNZdB7AmEl5NsyWrznRPz9ehhF0G6ljlfDi2rTCcEsOuj6wi9vfmDYO3HmeNZMke2uS dt5KHB9g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6UPZ-006yWo-1V; Fri, 24 Nov 2023 11:31:53 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJL-006p6j-1q for linux-arm-kernel@bombadil.infradead.org; Fri, 24 Nov 2023 10:21:23 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=tMMhe0jS9PrUVmoXP4Bu0W8PlaT5U2O4xT0LtFP/pq4=; b=o5feXquH7KkTMvT1mjQl3nvH/U nb1idS98yALtheDd8WTBAXK+MK4z3Uikcnsb7WRUPWmUfADh1D03dHHoJNRz5Zz7znoaagOjzw0Ip Suif1jHkHen89zydawd8BehP3cCuZQeBlTxriTJ4DrdctqDs02ThxwV3Y2v1X+T5JvGtrPmqjZFJB 3T5fVHbiryuUFHiwJG4tvpsenhCAA41wmAzxQBfmsCN1RZjQfid5TSLCCZhtSTOVAbR67PyRjdMIe nJ+T4fWsNv5xe90rGVbr8QpFb6THWjSdJOHNmldByAP8j88O90gE3zpB2dg0Qplx+qCC6shd2g/5H OCSvdOhw==; Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJF-00DqUC-2N for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:20 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-40b29868c6eso9188655e9.3 for ; Fri, 24 Nov 2023 02:21:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821275; x=1701426075; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tMMhe0jS9PrUVmoXP4Bu0W8PlaT5U2O4xT0LtFP/pq4=; b=QyLr8zbRqkdIO6mip8xLRQRi739nwjRj1IONFTJJAaqR1mkWDW6uzbc7m05fP+ifs5 YtjgawgOi/dLhpddrQi0QmE8wbzAgQH8ehr5CF6Q0Uc6B3sjuQZSMS8oLUbZ8+W2ipoz XqTXpDvYVHUrlXrDxenhoa7ir8kNdLvFAZVmyNONEKRqZpeKVUkr37mpRfNMAilfNKAC UN2k5VIr9oD5pM2+5BFuTYs8sp6ojMtw+69fmAechoXeqW2XHRmi8gKkJY3uAj8sImc5 mLu15onkeret6LBXP0J/eMytHTNjdE1zRRvdqJo4Lg4/5OMt/p5S7LjKRgCSRuoYe9Ds J/1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821275; x=1701426075; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tMMhe0jS9PrUVmoXP4Bu0W8PlaT5U2O4xT0LtFP/pq4=; b=F6HD+Pb8JkQDA2f0VtAOqMZpdGiNISVByCkOWs+/23My686pa2dbseGwyVxaY3DR2p oPcGzoMskvZHeexlaixU2zC9ZXppW+Ie/1eC5y/DKzIgBE1nrUcsxe4lrrR/Yyi+QnTg JlzrYHCzVeyGh/eH1OwE7tS8aTMUU+JMgorQtV6F7hAEGxKylDgi1rPD4Lgu6AWxifEz FKJMA8kvXpZMuk8jIcc06AZ9jL7FmRWZ0C1UhaBNswXyzNo6tg7gazz24hS6iK+0mvSl eRplfXNyx/BZuG3p7TIRpg5YcUJP2UuX1rU/qaD+TkAIxEIujxPMAkPwAJ2X7dcZi3Qq llLw== X-Gm-Message-State: AOJu0Ywj88vht64xjQihDTm4nZgVHtRVBfvUNGsqZlAPoD3tpCUvBb0g xt3LrSC+/X8vvx3/CvBpct8wbnzuheDDBDzBfsCOpJo+7cFhf/0bztbDTIsAKGSHm/UxScOIC5B bVVJZr1T9CvelODMyFsTo/OGkOl6f52uOowbNa0Q5qzCMdXDc9T3hxdWmH/Yr3nUnU9oh7wDcFi E= X-Google-Smtp-Source: AGHT+IEwSBAF2KnDUvM42UZYH38I+JCoDNOHq3GWhSGFOTO22pQa6hbMV0hxSPRCWA61GWd8zknbA4yC X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:1d1e:b0:40a:1f3f:be95 with SMTP id l30-20020a05600c1d1e00b0040a1f3fbe95mr46494wms.7.1700821275510; Fri, 24 Nov 2023 02:21:15 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:07 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1576; i=ardb@kernel.org; h=from:subject; bh=JtjBaspfCh1uG8h8kKWYAyrJQ/3eFGtzV80JlrHN+gk=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYpnTLP1Ic+2XS3sPLW/0lhIV/OheqDB32/K3ayT3F URrvXbqKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABPR/8fI0MIux/epMr6Xc8bk tC3290+WfYnTXOMm/XpF4LHTC+dmfWf4XxXE+D9qt5Xws5mKcybH280uWZXMEftVqUL/4+/Z63/ VcgEA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-68-ardb@google.com> Subject: [PATCH v5 27/39] arm64/kernel: Move 'nokaslr' parsing out of early idreg code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_102118_102053_D444102E X-CRM114-Status: GOOD ( 12.16 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Parsing and ignoring 'nokaslr' can be done from anywhere, except from the code that runs very early and is therefore built with limitations on the kind of relocations it is permitted to use. So move it to a source file that is part of the ordinary kernel build. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/kaslr.c | 7 +++++++ arch/arm64/kernel/pi/idreg-override.c | 7 ------- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index efbeb8356769..1da3e25f9d9e 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -34,3 +34,10 @@ void __init kaslr_init(void) pr_info("KASLR enabled\n"); __kaslr_is_enabled = true; } + +static int __init parse_nokaslr(char *unused) +{ + /* nokaslr param handling is done by early cpufeature code */ + return 0; +} +early_param("nokaslr", parse_nokaslr); diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index bcba0ce71af0..26961e0f94b7 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -198,13 +198,6 @@ static const struct { { "nokaslr", "arm64_sw.nokaslr=1" }, }; -static int __init parse_nokaslr(char *unused) -{ - /* nokaslr param handling is done by early cpufeature code */ - return 0; -} -early_param("nokaslr", parse_nokaslr); - static int __init parse_hexdigit(const char *p, u64 *v) { // skip "0x" if it comes next From patchwork Fri Nov 24 10:19:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 04F89C61DF7 for ; Fri, 24 Nov 2023 10:23:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=p+y9MjOGqazUYoaiAOrlIDRdmTxHit/9OGvI/DcOs6Q=; b=fMFQfExDZx6rgA5TZDw4CWwbyr YmflQTSXxxuq8k1Me9pYyAe2kyhP6TodQE2sXcCGHwDBlEl6M711VnPGhY1/1/Tymzlxj5594cpKW 8N5YggtVi75xqrxoqOg3CSyYqRChDoIw1Z+rOa5xVJ7Fn2jQgHY2SAHotYXqmV7X/yTf6mNaFKcl6 Z2bzPKblFJQffZxllYqvPBTYKblFYzuASSvqr4Eknl1iIEhM98xfgNnOQ8Ro4y3zK2cpAiCLavkAw hBVPl0fWiCu3aAo7FUvXpAMp/DzBYLIJqplGgaU0h6plK1u2sB2tZUU/pdOQ6iA4C79YVc9YIUhka OYpcyTnA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TKy-006psK-1u; Fri, 24 Nov 2023 10:23:04 +0000 Received: from mail-wm1-f73.google.com ([209.85.128.73]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJJ-006p4B-1S for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:34 +0000 Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-40b2977d6baso8984255e9.2 for ; Fri, 24 Nov 2023 02:21:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821278; x=1701426078; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=l10XqWeu41AoBxa5WcSIhVpgfupQiGyV50BGL3XWLaI=; b=hEsed0A0Ec346171CcstTmcHaBRu4JBTimRg6UfXsOrerCqPRhZYLla5sjL/CR6tzG B/J4SUhaXKDbf66WY+XBnHan81gGc29s+2LL4VydXtPsQZ0JjPkbd9luWe780cvj8hx/ OCKeW+VgNSht3eX2Una3pCjGIER1WjoxV9qstBo6SCeJFP8bYVAtgJYbuA6JsWG1jJSJ b0oBskNxVtNNDmaz0XCyONTx3T+UAtMjpoS1QQIJfsJD8p/+xwOvHdVljlcNEGtzcqZs NeCrKB/8kiJXUxsxOuamKa1Cvwq/G4tBhijWj2nEc/AXIeIgn9G+qAaQzxY09FBy2E0v B89A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821278; x=1701426078; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=l10XqWeu41AoBxa5WcSIhVpgfupQiGyV50BGL3XWLaI=; b=P2uQvXMTLZh1iYH5DZm/Nf3bb0ieHpk6EybtmsDNeSPoztTg7YycCruOX0SIvXFXrB ZZeEJBDH6E8ecy4AC5ackbIdszXz70Ie13pgPp5ovOhenHvVpkG8riN5K7QLK69Msv6l w8GJ0dFh2W+aQmYc6FxWkyiayHeKKR4mRvWsSWhNYRiG1zHtHYPq5Nd2tx5bVI3pJzVT l7SzRInD6ue7364Uw4Jh6QR6dqdZs+WVklIPmhJ/ofMKdWojg7tdPFdSTWZYDXV4jVz9 9CmUQdnJe+5f1rreGRfez2oNHedznR5+AR5PRXvyFvapckf+m4s0D/7WgV4oaC67N/Ww UsAQ== X-Gm-Message-State: AOJu0YwLTA5ehICbxHKj89b48ZzD7g8aqYlPff3sw0vzTgfC7iBftK0m fSZ/n/mgRBkasqot3MdBspCnoasD9itT9iFYHRsg2PLAW5IVAXYvPuOYY6y6T+RmgDKAV3tfvNT /PdDaCOa7TCPDiECo0giUTssSirzCpnQSnEzRKryE6+3fAnSeZPtrUeAxyLm1FRfWtgywEwDFIZ w= X-Google-Smtp-Source: AGHT+IFPwX748TpU2Rpu6cal1gywIaZHHP/ikEko66D19KgzmdCoSGw6MAdO5MALjZIG1AvZ65W8mRW8 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:1c0a:b0:40b:39cf:6db3 with SMTP id j10-20020a05600c1c0a00b0040b39cf6db3mr40416wms.2.1700821277946; Fri, 24 Nov 2023 02:21:17 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:08 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1650; i=ardb@kernel.org; h=from:subject; bh=5t7UnqSXZF9he1WJ0ftMHEdvuYDP8mp+eFRvxIU3k90=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYvmbx73X5hS4WG95wvC24q2935bMpfFhqV825W+WZ NTU2FnQUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACbitYXhN1v/LGs3tiP1i5ha uI/fiTr8/dLBU5N/pFqs3fk08XnW4nqG/76NnZEZaq8j0/aEuMwriDxUNdn+J+tv7XMr1quVuQm 2MwEA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-69-ardb@google.com> Subject: [PATCH v5 28/39] arm64: idreg-override: Create a pseudo feature for rodata=off From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022121_517089_B2C1759D X-CRM114-Status: GOOD ( 12.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add rodata=off to the set of kernel command line options that is parsed early using the CPU feature override detection code, so we can easily refer to it when creating the kernel mapping. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 1 + arch/arm64/kernel/pi/idreg-override.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 77ed3b28cbc6..99c01417e544 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -17,6 +17,7 @@ #define ARM64_SW_FEATURE_OVERRIDE_NOKASLR 0 #define ARM64_SW_FEATURE_OVERRIDE_HVHE 4 +#define ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF 8 #ifndef __ASSEMBLY__ diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index 26961e0f94b7..1aa59c01ab33 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -163,6 +163,7 @@ static const struct ftr_set_desc sw_features __prel64_initconst = { .fields = { FIELD("nokaslr", ARM64_SW_FEATURE_OVERRIDE_NOKASLR, NULL), FIELD("hvhe", ARM64_SW_FEATURE_OVERRIDE_HVHE, hvhe_filter), + FIELD("rodataoff", ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF, NULL), {} }, }; @@ -196,6 +197,7 @@ static const struct { { "arm64.nomops", "id_aa64isar2.mops=0" }, { "arm64.nomte", "id_aa64pfr1.mte=0" }, { "nokaslr", "arm64_sw.nokaslr=1" }, + { "rodata=off", "arm64_sw.rodataoff=1" }, }; static int __init parse_hexdigit(const char *p, u64 *v) From patchwork Fri Nov 24 10:19:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467458 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9038FC61D97 for ; Fri, 24 Nov 2023 10:23:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=l2W4h7wMYMQcl9K0HjdxfxAfiiqPt//zePEvCNjvpxE=; b=ZdNNGAhpKE/ckLDRtCD9xyIuvO zunuPo0PSn8gnN0Z8R3xDifQXNoYQCzhz/86GgwRKPlnimz2/O2JLQwzhCoe5amR2nASoikQnp3BN AqibJIj6j4fcD1p2dZ8z3QZ2Od4TjT6NSSKMAmKhHO1FYROO12qU+lVuNpDxXaeT+YhRCNDk24Uki ImYhE4nOnQ34LIZ3QQnaLwFBgzGY/QTATafu+EUpJbmhPQKFUopzuqI4GKDzd66uQhOkCyyyjsMIp cMpy2fVPnYwwa0UlQeqZKiQGqIxEjE7cwDFeGUBTLHqFxlgy9doJbOPmhZrKW/rMveNI3GkZWDHWg BTruo6FA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TL1-006puI-1w; Fri, 24 Nov 2023 10:23:07 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJb-006pJ2-29 for linux-arm-kernel@bombadil.infradead.org; Fri, 24 Nov 2023 10:21:39 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=mszekfn1Sv+AM+mXkqTGdx2TrBZeMlZMYtBcAH7wCcQ=; b=lUF9EsYK8m0qLAlXLIMIRW8K6h iapCi2bszQY1xrA+aoCNnwN3h+ducMjSwPptag4uy9HL9SoApj9G4ZycD0JCFp3HEAEWQeF4fQtJi XYAETzuAhflDDT7+MFW81t1x362kLlKU+cgWhrKBurXG0nFXNnn2xa4y4pPr4scxE30BH8UBJBx1G +oBp9hygkNXsa/Yo/FRcMlen9Fzl4g+hw+IhXDIibaJDnr/h0lgpe038TopmJuMTf1NHpBqKjtYbF ouYgHpGf9vd6E4IdpFJlTIsbGDQHLilgo9VMCAUQWAHbRo9Z4gasC0WbEt36xwprZA6ycqng+WybD 6tbKs/0w==; Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJM-00DqUt-1d for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:34 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-332c537e648so923216f8f.2 for ; Fri, 24 Nov 2023 02:21:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821280; x=1701426080; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mszekfn1Sv+AM+mXkqTGdx2TrBZeMlZMYtBcAH7wCcQ=; b=iERSqAr8FD5rOYG0AGuSuESOZaRNf6EKQTlUP0i0ntBV6Yt3QNukcHvCIQNKWWRT0z 0xgRuD2OAyeFhgTMLenhSYPExlRM45frIYHRfi4VuvviFM+JXW8UYTfmn7Vk8L/B+VkC zG1uy9Q3D/M0y2lM4vaCvML1d6zfXeDiQfSc7dsSSIYZqGStHeSqb7viS4c0toBKgYNB V+KA0QlkBA0BD6kI2UKYgD/UHIQ10/4pY6fc6UqHy/aptaNxs1uwKKHv5OacyPMjT+3U pRqmJEBs3w9ZxxjfKvNKqUtFzkE4vQ/QcSNDhDNP+f4Fhnch+v8E9EBsPnb3z6fHlzZp 5UMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821280; x=1701426080; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mszekfn1Sv+AM+mXkqTGdx2TrBZeMlZMYtBcAH7wCcQ=; b=XivdUTKW9cmIHmUZUru3CxXNbuaK6vtXrfGopyas4ZhKs/naeVMZcfJlfoCBoK/eRi m04u2ozGSni2oZUyMxBxxN7bX836JUTmXQHVO6ftSuYtYQ+ztVf1Kl0o4fUx+lKEvhal ZceIw91qnk4SIjbx5oB5qXYB/feqJNNTmXUoyZaYyw2Z4tichZv3Oy1TAfdxTgmSS/lT vGReXNBiRdzVqZCY4A1TPAVdByeDXh/IX4vYF+fQFAVmKwFzEFi3KI3Lemndn7hILzm6 KBlfvkQYA5qgre7DZ+gezHU6MPFiW4MYSZI/pKH3Y/kxk4eLnSpIgv3JlUJiyTodBV/t 1BLg== X-Gm-Message-State: AOJu0Yzq0LZTnU7afZXEOuUlIPkSE39TTXJ508DSfbgM4Zk6ZgC8nsrU U509xjWhhOqPVVmqBhPtCo2EVWRBgCkURIC+SUjS4jQS2Fx9pVrW/g9y/9Tb3ChRpyj8GUBGkz2 YVyE5SAt/L8tdRUJizJFyX4p4BrvK0fQQEta8BfAswYZLt3NfW+Al3ffMkLPjX2zJFEmuzG6G2P w= X-Google-Smtp-Source: AGHT+IH3M2MAbdE75mZzRcufrS9P9LQ2paskEZja/Tzt0+hvF2RxW7j8+v9DvTgPty24/FK9QpyFWKt7 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:247:b0:332:eaa7:56a8 with SMTP id m7-20020a056000024700b00332eaa756a8mr17450wrz.7.1700821280115; Fri, 24 Nov 2023 02:21:20 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:09 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1939; i=ardb@kernel.org; h=from:subject; bh=ojuDpQogaCV9dGBD05qcTAXvoBAjXo37UNGpggV+Bbw=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYiXH2btfJysXhy+9eU/E93XE9cvhgdp9L5nWNq25X H2248ncjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjAR+VaG/6Guy5peXL0TFfVP ad3NYvmVS0qkHHYv5E/Qs+69zyAqwcvwT2vZCp1zf/Zms/LFTw+7/E13pk7/67rn36eUXry2/6v FVAYA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-70-ardb@google.com> Subject: [PATCH v5 29/39] arm64: Add helpers to probe local CPU for PAC and BTI support From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_102125_476650_8464DCD7 X-CRM114-Status: GOOD ( 11.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add some helpers that will be used by the early kernel mapping code to check feature support on the local CPU. This permits the early kernel mapping to be created with the right attributes, removing the need for tearing it down and recreating it. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 44 ++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 99c01417e544..301d4d2211d5 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -921,6 +921,50 @@ static inline bool kaslr_disabled_cmdline(void) u32 get_kvm_ipa_limit(void); void dump_cpu_features(void); +static inline bool cpu_has_bti(void) +{ + u64 pfr1; + + if (!IS_ENABLED(CONFIG_ARM64_BTI)) + return false; + + pfr1 = read_cpuid(ID_AA64PFR1_EL1); + pfr1 &= ~id_aa64pfr1_override.mask; + pfr1 |= id_aa64pfr1_override.val; + + return cpuid_feature_extract_unsigned_field(pfr1, + ID_AA64PFR1_EL1_BT_SHIFT); +} + +static inline bool cpu_has_pac(void) +{ + u64 isar1, isar2; + u8 feat; + + if (!IS_ENABLED(CONFIG_ARM64_PTR_AUTH)) + return false; + + isar1 = read_cpuid(ID_AA64ISAR1_EL1); + isar1 &= ~id_aa64isar1_override.mask; + isar1 |= id_aa64isar1_override.val; + feat = cpuid_feature_extract_unsigned_field(isar1, + ID_AA64ISAR1_EL1_APA_SHIFT); + if (feat) + return true; + + feat = cpuid_feature_extract_unsigned_field(isar1, + ID_AA64ISAR1_EL1_API_SHIFT); + if (feat) + return true; + + isar2 = read_sysreg_s(SYS_ID_AA64ISAR2_EL1); + isar2 &= ~id_aa64isar2_override.mask; + isar2 |= id_aa64isar2_override.val; + feat = cpuid_feature_extract_unsigned_field(isar2, + ID_AA64ISAR2_EL1_APA3_SHIFT); + return feat; +} + #endif /* __ASSEMBLY__ */ #endif From patchwork Fri Nov 24 10:19:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467452 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 29335C636CB for ; Fri, 24 Nov 2023 10:23:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=mK3JVFzBvYg9OZ1Cd+BGQpHboVXWZqaU0x+d8g0SKKw=; b=PmcwyviW7HyNRYXrE4tlaRF4Yn pQiJL8E2FBK3UyZxiv94tAo/T0rXgyN9r/2BeYblVeNAD8THH61/XQ2/Oaq7CLNzT8GyCKsep9zja fk+xpdS+6xucD2WospOwArR31TCyUAu58Xac5Yeji4S6prZOtenM08qlLeOX7gX7orkpHNyb9ZgXu CUpi1EcfRjFyXQFEngofhPv4NLEn8QL2iZMPwEHM4vSCwJnARYmQ7E6/+k6peM8/ed9+odEUXk/GA 02lESZ7z5NG164l7tUlj6iGlD55B1bKnsJpTKE3KmrVyJDymcrABwwUKGrKta5EQB0n8j7g5R+eQL dvZCoEsw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TKx-006pro-1k; Fri, 24 Nov 2023 10:23:03 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJU-006pDn-39 for linux-arm-kernel@bombadil.infradead.org; Fri, 24 Nov 2023 10:21:32 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=RRCI6wTgDgs6zQ9NwIFImncmAA+W9fXLCfXKh6mbpP4=; b=I2+F784w50At1U80Ml1Cqmmtbx wb/GXs/1XZDHrQ8OgGhMnpUruTPGqgSoqWUCv4esyhYB7rVdbD48xbEJV8meHT9RQkEXGYPmEwCpu 03lQBBvY4M4+1pjBOEzgFBUJ3kxZ3ltCn60WVft+oIKrxEY0dSE0EPl6DjDl6SL31JYjIdbAQ3hyT jmDTHMSaJA9eDl2kxXZ1ZR9s6T0GBFq67KrLqg/+prWSS+CGr433Pco7p0IpG4KUccer4yc0xuKDr F5FNCos9803jcLjTrRJ3AtDTUifEuLfs4IGS0MTD08vrPAxA2qVW95S8+mcjWyqUoIoc/4ZwfDIJC wGgAUSLA==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1r6TJR-008Sq7-8F for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:31 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5cd48fcc2f5so18993617b3.3 for ; Fri, 24 Nov 2023 02:21:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821283; x=1701426083; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=RRCI6wTgDgs6zQ9NwIFImncmAA+W9fXLCfXKh6mbpP4=; b=iwgFcP/KdIJ9INHQiLIxl/r0ak/GIgelITcmRypfxGdSve1OKsSJUFK0IPx2XPQSl7 ubKj13qMHzUhA9DraeW3Sc3OmL3kFd/fcmffhg2uW9HBocaue4feCExmx9M0GKOh53QG J/lqoLBUKlk/nDWjbSNeNbygRTfZOlDA5BJg3fDiYkwaYtE2fRCxON918LvYeiLBVBOP mp+YoffQYMfu01wqn+AzTW2Hua4WY/59soXzvBz4ICAuKEXN6tl68Tj9ocZf0p23g0/S sAcpE30rEs5oGhg5XO/8NG3+0NZ1L5DYELjV+sx7fI9BSKfbz7UQRbwYqB4TtdNVNGIL 3BQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821283; x=1701426083; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RRCI6wTgDgs6zQ9NwIFImncmAA+W9fXLCfXKh6mbpP4=; b=m+Ggrvf7uQrkfeiW78pqoLWylo1cQwQsGJOgIIeAkvzL2e8AI7Lzv+CLFrLOxBAmQU RYT0yeQsfY73hPPjscwXfS9RiAynRuuD96EpzvxBRB3pxHApnNncidPnA5tj6epLAlWz qxNijDLuuiBfBvhcb3Aj9qe282ctJRp/su8g6z1HeYRObTpIQfPkHeFP7Hrhr5WwrOvz jmTWTkPnfQLhUnrjpj3L+ppM+8goaXRbX/MpWrJbkozO4zye898EpSGataa3iX/5eIPn aESFEwrKfMNCcqrKlYq4eTvN/MbTGdWGy+hPforli06UO33nGYDuv5oMW0kQmUuzlJHS sz4g== X-Gm-Message-State: AOJu0YyedsjnPgCXVCtRyRpRR0UfczLzkaCm0g2S0EiEvp0zECbmQrtg 3NLBVbYcXGr+LJOzvGHpP8i0Sc+/ybm6odYW2gGetdC2PAOVWADswaB+eWTlMpnH5keCGCYztZu tYj8LtqCMQk/COwg82tTJmOuJ/pH7T3zzh+DAlJkZXYVSRYFUxdG1OEtENDbjmNLxICaKDMlkks U= X-Google-Smtp-Source: AGHT+IHI+A1ooh17Jn9Lp63eEnO4gZa7L7w1kTwy2YSCvKtom09weJ1m5AlmsPMyoY+w3wMuOKOPq9gl X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:690c:811:b0:5ca:f8cb:181d with SMTP id bx17-20020a05690c081100b005caf8cb181dmr70142ywb.3.1700821282661; Fri, 24 Nov 2023 02:21:22 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:10 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1838; i=ardb@kernel.org; h=from:subject; bh=CTH3x4jA7y7C9opFImzsJKcifiJL/bHaXF8OlzVDljw=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYlXLB/P/uX/Wr9P8+eq96ML4bsM25xSR39I+0/ptp 78oXtbcUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACZyr4Dhf/bzdXPWsf7NOjv9 0d9bzx0vPK0vPcq8c6+Fbpm1Vn1B7DmGfyqsH5cmr3Z78vDgRPEc5ogiZZcp5+beydLr0a3wfqn ByQcA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-71-ardb@google.com> Subject: [PATCH v5 30/39] arm64: head: allocate more pages for the kernel mapping From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_102129_379034_29C52AE3 X-CRM114-Status: GOOD ( 12.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In preparation for switching to an early kernel mapping routine that maps each segment according to its precise boundaries, and with the correct attributes, let's allocate some extra pages for page tables for the 4k page size configuration. This is necessary because the start and end of each segment may not be aligned to the block size, and so we'll need an extra page table at each segment boundary. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 83ddb14b95a5..0631604995ee 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -68,7 +68,7 @@ + EARLY_PGDS((vstart), (vend), add) /* each PGDIR needs a next level page table */ \ + EARLY_PUDS((vstart), (vend), add) /* each PUD needs a next level page table */ \ + EARLY_PMDS((vstart), (vend), add)) /* each PMD needs a next level page table */ -#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE)) +#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE) + EARLY_SEGMENT_EXTRA_PAGES)) /* the initial ID map may need two extra pages if it needs to be extended */ #if VA_BITS < 48 @@ -89,6 +89,15 @@ #define SWAPPER_TABLE_SHIFT PMD_SHIFT #endif +/* The number of segments in the kernel image (text, rodata, inittext, initdata, data+bss) */ +#define KERNEL_SEGMENT_COUNT 5 + +#if SWAPPER_BLOCK_SIZE > SEGMENT_ALIGN +#define EARLY_SEGMENT_EXTRA_PAGES (KERNEL_SEGMENT_COUNT + 1) +#else +#define EARLY_SEGMENT_EXTRA_PAGES 0 +#endif + /* * Initial memory map attributes. */ From patchwork Fri Nov 24 10:19:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467454 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6EDD8C61DF4 for ; Fri, 24 Nov 2023 10:23:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=/P8uAQwn6ZJM54pinxAGNr8ZlcOcTFSdkn5I1gAz50g=; b=Lcj9ylN8wgr/DlzQOuodyPbGDj CT/dOLhSXRXWDBlV5xsSF+Cp93Zgo6UaAadQPLIgH8FN2GgaXNox01T5tZaOthVLlDie9fZGeDXGt qxtArc+NZifeiULSbsST0yLwkr8O0X6BmMJ8yrP5gbvmeVPP6TTft7uWuWqv9yvxS0Q8MTruoai0u n8DxHm/4FB71Y5cxhn7xYYEYKMcmdLgFq0+KXS5jc6t+N99jruA331QZx87gemDnaadd3lJhzV85n Z+TWvrNDDYNrAuz7jmS+rSI6ESA9fuZzPZS3vD97CVnf1L1IT5IThaBQYKDUuswQKt6G3aOeR35mv ncU4OHDQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TL2-006pug-0Y; Fri, 24 Nov 2023 10:23:08 +0000 Received: from mail-wr1-f74.google.com ([209.85.221.74]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJQ-006p8k-1z for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:39 +0000 Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-3316945c6e5so1485869f8f.0 for ; Fri, 24 Nov 2023 02:21:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821285; x=1701426085; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8x+lhCJ8yCfvO8YybIBpJCwOL4gY19sphz3/zZeACpQ=; b=TAyaSDMwgI32vGLv5lOHC4caSN3XuP4nlEfse7m16FxJwokioNry8kAZU8wwqgy8Fw RABbCjxXDbeD+8xHNE7P+Wgb8P+NZXrMYcy9YMYVpY4eZD1hU3Cjfyo1kPefU6fxL6N7 ma1SbV1noyVNMMcy5LGhfQXKAJn2SCDGZEZ5GBY2/4LwEssH+pAhyq44n5lfLJXnQAEr VTkJ634EhXuHbqNF7gGlfsFs7GeEVF9WmHyLktdGDt80uoCNgjWz7V9rCr1YkBzaNDRq uq/lLJbp1YsHKjU0LoN2bjotq7xbDxIjj574FiXnKE/cdH+ZH1yr9a6O3yC3TveoJsx6 ug4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821285; x=1701426085; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8x+lhCJ8yCfvO8YybIBpJCwOL4gY19sphz3/zZeACpQ=; b=D0eTHjjryo/LDJuk8ssPRTfUne82csUPzNPrzWKDrRL5lVECj5bLOAQ1Bx4ATJZwHc rbQ/rrgR4xCtC7mp0nfzAnpdmKrSWTItVa1M08Ds1+z9owYJTByj95R5RAAc+8s3Rgbl jX7RSsQsV8kr8z3QH2E0tkTqpVMpLvIJ+WgPYY2XefPY/mlNxDN91AkBIo6sz5SIehtC yeeaNzPafvIisC5Xt1XlViewyaRiGRDdUDGzFFji5AwW5DOsce2qec2/wPYnv9Bf2lHX yMljmEtWDN3/oZwJrTwh2qVaZN+PHyNJbmBBkPI6tQ1y/FHa4fVJahFTPdIeLUL4q6Ag tsUQ== X-Gm-Message-State: AOJu0YwSoIdDSRS5H8jGVIsC2qfcRxWI5K7IPIoCXV6OAR4doOaYWSd9 2LKaLy05zgz75EQptDITJMMQHCFFN/J4RIjf+oYE6D2dZRVkzilG2PbcaaiWsl3P7gw7JAB/zzj E2LZgjoZ8duZ41lRkmqUqCUu0IrCOOVqGQs8TLzwqRcBt6fzhmxTZKMjwu+m89/GcuQvKm2x/Uh 4= X-Google-Smtp-Source: AGHT+IHsuBJyBn5SXiKjAa3MO4njVIWy4MrORTDuPrQ7uTqDfVvbe63HbR4np6YuoBiYd9OFl7kw/OCY X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:eaca:0:b0:32f:997c:bd2a with SMTP id o10-20020adfeaca000000b0032f997cbd2amr90460wrn.5.1700821284765; Fri, 24 Nov 2023 02:21:24 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:11 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3181; i=ardb@kernel.org; h=from:subject; bh=rA2xvJle278NGePfBuK4ULUQimFQTpGFGVXHAgHwJXY=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYrXHvYk7J95ICpi9zGrXus/9nnJPO41PzuOXuiYc2 Mla+mlDRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjImVKG/4WXMq6HXbt/3mjq duXX/s+1KuouzFrL8/0tq1gJXwbnfm9Ght9HHa6Fm7NonY0u6zFNKfe+U8DdunCKSWjMyb0piQ5 FbAA= X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-72-ardb@google.com> Subject: [PATCH v5 31/39] arm64: head: move memstart_offset_seed handling to C code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022128_671738_C16B0F75 X-CRM114-Status: GOOD ( 13.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that we can set BSS variables from the early code running from the ID map, we can set memstart_offset_seed directly from the C code that derives the value instead of passing it back and forth between C and asm code. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 7 ------- arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/kaslr_early.c | 4 ++++ 3 files changed, 5 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index b320702032a7..aa7766dc64d9 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -82,7 +82,6 @@ * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob * x23 __primary_switch() physical misalignment/KASLR offset - * x24 __primary_switch() linear map KASLR seed * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register */ @@ -483,11 +482,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) str x25, [x8] // ... observes the correct value dc civac, x8 // Make visible to booting secondaries #endif - -#ifdef CONFIG_RANDOMIZE_BASE - adrp x5, memstart_offset_seed // Save KASLR linear map seed - strh w24, [x5, :lo12:memstart_offset_seed] -#endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif @@ -779,7 +773,6 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RANDOMIZE_BASE mov x0, x22 bl __pi_kaslr_early_init - and x24, x0, #SZ_2M - 1 // capture memstart offset seed bic x0, x0, #SZ_2M - 1 orr x23, x23, x0 // record kernel offset #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index eacc3d167733..8d96052079e8 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -46,6 +46,7 @@ PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); PROVIDE(__pi_id_aa64zfr0_override = id_aa64zfr0_override); PROVIDE(__pi_arm64_sw_feature_override = arm64_sw_feature_override); PROVIDE(__pi__ctype = _ctype); +PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); #ifdef CONFIG_KVM diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index f2305e276ec3..eeecee7ffd6f 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -16,6 +16,8 @@ #include #include +extern u16 memstart_offset_seed; + static u64 __init get_kaslr_seed(void *fdt) { static char const chosen_str[] __initconst = "chosen"; @@ -51,6 +53,8 @@ asmlinkage u64 __init kaslr_early_init(void *fdt) return 0; } + memstart_offset_seed = seed & U16_MAX; + /* * OK, so we are proceeding with KASLR enabled. Calculate a suitable * kernel image offset from the seed. Let's place the kernel in the From patchwork Fri Nov 24 10:19:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467450 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1538DC61D97 for ; Fri, 24 Nov 2023 10:23:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=+kUihMkkXRgd8YdcUQ/SGVZqG4IttXDOz1CT6Ey9Lqk=; b=g4Sf1VI+rEH9a7qcG0Mougro37 U/cFc7M4nKVkaRTkvosws7DsOi2pTvBshCrpIAqIlA37rzFbVXGfPEGUSz1IcuLlvqFDlsZkweoXP UICUx88ZgL4ly+WU2nsSl9oJu0LQEGJgbhN6BG9tUVjtl1I7dnBWK4UX9gERLI9c1so2ICwrFcQwD 6mUwJb4ipfyYVES9R2g1ibDRY8i8zxjhYRjIs2GRWYEV4yU9fLlrzY698jO2QBG9/6A2AUj+a90OZ am08QpM752sWkfYXRa5cdePje66VdhlyYLF4wx19F3nfJm0koqdvZyJBYl18B/2IEmnJVJ75FWSDs GpWKAypA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TKy-006prt-0G; Fri, 24 Nov 2023 10:23:04 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJU-006pDo-3C for linux-arm-kernel@bombadil.infradead.org; Fri, 24 Nov 2023 10:21:33 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=qVZ3aLKPTG+oZezVT42SNwhH7ArWgZIhCYiS27q7XPw=; b=s7WFY3UOOQ/1ZUJDpvj3hTnTC+ xxZJ/pcLZJ77hGoPqpuE5XspZFJnl7WnAwd27JAqBbz/yGDSqZTgHphL/qcW8GzD9NvLGkslFqp7k /ijoT/LWR7XTJZ5SYz80JFfqsEG3v07+w3iP+gXmJhn+Gs8PElDstoQe2u45CEafaJuvnGSnxTzTs C5V26T1VziSophwDR52A4GjF0lyVs3JrMvCxHi/G+CpP7UU1iR7xPD0CJQOKrYCbR0+YRDwhcSAig 8TVyXqlG/VJMyrK3o41afKTGBtHtC3KWwEDzt5i26PQZm6qDVMMVQGceGObgHRjRbZ6dyQJEawWXk EeIAEwUg==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1r6TJR-008SqN-FN for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:31 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5cc3e334d16so31260477b3.3 for ; Fri, 24 Nov 2023 02:21:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821287; x=1701426087; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qVZ3aLKPTG+oZezVT42SNwhH7ArWgZIhCYiS27q7XPw=; b=rLy4rQCkYdzpebJ98DSV/pfkxmsrsvu3HxE48JxJncnUrhQ/xEsgrdsyEI3q1yZPsl hZed/TnM3wKrG8EadMliGupOQ6/NgTN2rFsZtnRe/wcJG4nJyFy7UNlyZJM4p/ioYOs6 tOGn3kzMEZ/rsyIdmQI4T2z2nXrKEHc/Nwkl+DYCQhj7l0fNjK2ZjasJ/NukR+ADwA4p bRrsfQQLpgsJK82xX3VrHQX6AGBNJowUYvyIB4t7F48Hgl6kw9QM6U+9OjFE4nsOn+Dt Q35SJHxCrLNqPCbslLIRwvFaWIg4G6LA2MgJQshKYBUQgCo8qUNCfID93ImX5SA0/N97 hirA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821287; x=1701426087; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qVZ3aLKPTG+oZezVT42SNwhH7ArWgZIhCYiS27q7XPw=; b=nK9uNX3un8k1A0bUgoCvapx+tlLD5W0RzMBiMqjlUE9zD0STzM99b/zu7+rzuxqn9D LcqWRcHXsKzaKLsQAF8L0fm223dM4I0r0/qUvQ7I2TQzbm3PoTFsOck7C6NGy/TeluH7 sFhZgsLPjVG8R7aP/WrF7jKQo9eID5K5gEFqA1AUCOUXgLnxXxhahKIBo6N7eOgvkBG7 +lZcGqeDyb6+qqIx4T4DsqsBzoRlMlppDxmP69zCpB6iQlKjMAGik/nspSSB1pcFuOwl udW8YosSQgD+KQBumAorXlYzIbudiF+V9SqlmQuZJx0Lk18H598FSlcIWfX9/RDr7WR9 Sw3A== X-Gm-Message-State: AOJu0Yw4copW4JElJgplc+KfQ9IpMIfJ2/KTizwvodqOkT44NNkk1NPC w9diESJZm/0HTmQh38v30eVfFsK1xDuObogAk1vLJAPWU0uLPfx0SHQCynEdPNmHHmyKTh0fesb nbT9DbDjIBCpid8gvj2lNszYOs+EthFxro9OVY9QmxQGOLsv1gfxTi+I922NDM+Rn+LIWSv1CYf g= X-Google-Smtp-Source: AGHT+IEr4y3t8vkSbcL3DHVY7tO8qbgIK2id3Vx6CyENpCFACgH2jhxdMXZ6P26swyPOVK18swa8Eb8k X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:690c:3102:b0:5ca:8979:e650 with SMTP id fb2-20020a05690c310200b005ca8979e650mr61676ywb.10.1700821287038; Fri, 24 Nov 2023 02:21:27 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:12 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5079; i=ardb@kernel.org; h=from:subject; bh=9eDZ1Je5zVxggBBmH3/anCK0BzsjoAP/PqvFC9cVPlU=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYo2dS/q6b8f5GI0yA36pfuhpXRx1NDH3tUlv9zbbs PQTyyw7SlkYxDgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwETuH2T4Z2xvbFLB7RVValbn tXLXG47jQSurnh/+0FTlaNkplHB4G8N/f+Zmd+3aRzufSfUHzReduKla9MJThfndD7lWZz6837O PCQA= X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-73-ardb@google.com> Subject: [PATCH v5 32/39] arm64: mm: Make kaslr_requires_kpti() a static inline From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_102129_550133_74ECBB18 X-CRM114-Status: GOOD ( 22.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In preparation for moving the first assignment of arm64_use_ng_mappings to an earlier stage in the boot, ensure that kaslr_requires_kpti() is accessible without relying on the core kernel's view on whether or not KASLR is enabled. So make it a static inline, and move the kaslr_enabled() check out of it and into the callers, one of which will disappear in a subsequent patch. Once/when support for the obsolete ThunderX 1 platform is dropped, this check reduces to a E0PD feature check on the local CPU. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/mmu.h | 38 +++++++++++++++++- arch/arm64/kernel/cpufeature.c | 42 +------------------- arch/arm64/kernel/setup.c | 2 +- 3 files changed, 39 insertions(+), 43 deletions(-) diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 2fcf51231d6e..d0b8b4b413b6 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -71,7 +71,43 @@ extern void create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, pgprot_t prot, bool page_mappings_only); extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot); extern void mark_linear_text_alias_ro(void); -extern bool kaslr_requires_kpti(void); + +/* + * This check is triggered during the early boot before the cpufeature + * is initialised. Checking the status on the local CPU allows the boot + * CPU to detect the need for non-global mappings and thus avoiding a + * pagetable re-write after all the CPUs are booted. This check will be + * anyway run on individual CPUs, allowing us to get the consistent + * state once the SMP CPUs are up and thus make the switch to non-global + * mappings if required. + */ +static inline bool kaslr_requires_kpti(void) +{ + /* + * E0PD does a similar job to KPTI so can be used instead + * where available. + */ + if (IS_ENABLED(CONFIG_ARM64_E0PD)) { + u64 mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + if (cpuid_feature_extract_unsigned_field(mmfr2, + ID_AA64MMFR2_EL1_E0PD_SHIFT)) + return false; + } + + /* + * Systems affected by Cavium erratum 24756 are incompatible + * with KPTI. + */ + if (IS_ENABLED(CONFIG_CAVIUM_ERRATUM_27456)) { + extern const struct midr_range cavium_erratum_27456_cpus[]; + + if (is_midr_in_range_list(read_cpuid_id(), + cavium_erratum_27456_cpus)) + return false; + } + + return true; +} #define INIT_MM_CONTEXT(name) \ .pgd = init_pg_dir, diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index b95374387946..7222f5a14794 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1649,46 +1649,6 @@ has_useable_cnp(const struct arm64_cpu_capabilities *entry, int scope) return has_cpuid_feature(entry, scope); } -/* - * This check is triggered during the early boot before the cpufeature - * is initialised. Checking the status on the local CPU allows the boot - * CPU to detect the need for non-global mappings and thus avoiding a - * pagetable re-write after all the CPUs are booted. This check will be - * anyway run on individual CPUs, allowing us to get the consistent - * state once the SMP CPUs are up and thus make the switch to non-global - * mappings if required. - */ -bool kaslr_requires_kpti(void) -{ - if (!IS_ENABLED(CONFIG_RANDOMIZE_BASE)) - return false; - - /* - * E0PD does a similar job to KPTI so can be used instead - * where available. - */ - if (IS_ENABLED(CONFIG_ARM64_E0PD)) { - u64 mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); - if (cpuid_feature_extract_unsigned_field(mmfr2, - ID_AA64MMFR2_EL1_E0PD_SHIFT)) - return false; - } - - /* - * Systems affected by Cavium erratum 24756 are incompatible - * with KPTI. - */ - if (IS_ENABLED(CONFIG_CAVIUM_ERRATUM_27456)) { - extern const struct midr_range cavium_erratum_27456_cpus[]; - - if (is_midr_in_range_list(read_cpuid_id(), - cavium_erratum_27456_cpus)) - return false; - } - - return kaslr_enabled(); -} - static bool __meltdown_safe = true; static int __kpti_forced; /* 0: not forced, >0: forced on, <0: forced off */ @@ -1741,7 +1701,7 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, } /* Useful for KASLR robustness */ - if (kaslr_requires_kpti()) { + if (kaslr_enabled() && kaslr_requires_kpti()) { if (!__kpti_forced) { str = "KASLR"; __kpti_forced = 1; diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 4b0b3515ee20..c2d6852a4e0c 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -288,7 +288,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) * mappings from the start, avoiding the cost of rewriting * everything later. */ - arm64_use_ng_mappings = kaslr_requires_kpti(); + arm64_use_ng_mappings = kaslr_enabled() && kaslr_requires_kpti(); early_fixmap_init(); early_ioremap_init(); From patchwork Fri Nov 24 10:19:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467459 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7240BC61DF7 for ; Fri, 24 Nov 2023 10:23:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=uOLh6E96/eGMpMld4UUyGzdoeUcuIC8wmpPh0VxBECk=; b=ZQm7C1Ps2f3dN5t3JDAqywbdDs xYP6AREkqIUoW5d5K7eEvcvN/E6U3yUDfViyzMVRmZBIXdBO9wiEoJly1jIHPtVa1R8ysV98NZ0Vb aI06m660KTdi2DSX6SAhYR6TSjMUC7aENRRMT7xt9TYUBLxByuVrxSZIV2JWtpd81cEJwQrzhCeZV IZfQ5T/aCGrnsVrt0qvGYffbDUFclXst3TT07QNMAVpksyACdR0kyoWZIRsoElellC/xnRYt7fXYr Gb79NVU+B07hT3/gvkBTc+ESrGXh81dGePEdzqsHscyYG2UzMOdQGMZmoxRbpPCxuwjFPg1lB/vcQ ZXgka06g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TL0-006ptx-33; Fri, 24 Nov 2023 10:23:06 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJa-006pHk-1F for linux-arm-kernel@bombadil.infradead.org; Fri, 24 Nov 2023 10:21:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=WKm2QsxNxifj0O69/LUZMMqEzU5cv/jVGaCzDfAVuGA=; b=r96C3KZoebZInJxhUBjpz44DFB 7GXSGf7K2lLFZXjFsWQrBrThGGDzw6orMtwkVZ9xswk5pbQXckRZJoXScWpnSfAb9aai9BiYMDlpD dOCzsyzygckJVwjNwGboyHzdFIcHXfKZcg+eMUCFSYNTo6sYV6u/NnPrvvq62XQN964AFTA0QtIVm nrq82BwG1i3sQhHO6DCDe+ZO6W1opI5ctZ5plKeL7OmK47tiI9nueII8bsY8Vz0Rt+9WOUItD4osG CH8jt33NR7JXhOAx1/Ksko/JlKrCvu4b93LAn7Nhd5d9zyLjS0bPVLYGHVY8R4Oo1fmm8yhO/lAOU PCGGpY2A==; Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1r6TJT-008Srg-Nt for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:37 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-40b3d4d6417so867175e9.0 for ; Fri, 24 Nov 2023 02:21:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821289; x=1701426089; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WKm2QsxNxifj0O69/LUZMMqEzU5cv/jVGaCzDfAVuGA=; b=eJRbPNK+KR5zYQp5Vqv57HmRx4wadVV2MkgxTFEGnI7KfUOVKRHelR4YgRIVgT8Edx jC9NW0n9lM0ez/ua6jIojumssXEFJjboxN0uvkgAfrMG5AJMihfnP2J94TbB5FC731p7 2oVu5Xm+eQSTxCIkGgxI8VwjyiTjeJCxty3GjFY7iFpoEwtYJl3L5qNkJ3M1hHeJAp+r mW7Fh+Ct7GzrJaCNVXXgMf30GW508EqXwa5Zw1EnQlpFCf+I6Yz+CQNbVv14qB+EenMw EAHfRoRQOey1lM/SIrOHHknvW0XBf8+MpXxdJZUKvzjzSEHGEYKv9ftlPs49+PIp/zsv KLIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821289; x=1701426089; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WKm2QsxNxifj0O69/LUZMMqEzU5cv/jVGaCzDfAVuGA=; b=rpJo8nSvCkqLh70XLsWyg3s7CW4y4gf7IiOK086JobYiiXanKeRUG8Rig8UqNJq2Z5 AMlyBaOn9j2DUGoGfSxxIVMVG659R0EAYiOXFp5OKUTBKW+N0ueLwZadqy8lRSb+520C CxUHpR8mbJnOqZGoIkc0rg/0381vVDaGw8+ciLhSi6CYpfYc7XGaF+cIUFT4cIj/EUAC VsUDU7FijxULGYXH3RCP42YBDgppBE2zCdNlJLArwJ9k/THfGj1lKBNLMl5+WXVRbPxM U8ZiZujhfxdd/3Q+wxkSWuijSwinUQdM24702bd1bFp+WJBInp1UCvKr5oogpBoHEXiB HgAA== X-Gm-Message-State: AOJu0YynkzThRDakFgr5bpI2X3YPPAmNwaR1rp/jKxEbDuLSyAslzr+4 A7f8BizNN3KeDmv02uRvOZ5Co91Jq2COnxGC8LUjxszbl62QS/OPgkVIhoTrny4dGanDDDH5qui 2h/DHi8icAtgyc/eHkQ4sZf/fKymtgUYDi6MhPIBbHkmA5Q2S2MmUhUIANJybHjbPElROGD/Bgp E= X-Google-Smtp-Source: AGHT+IFJx6020TQjBqu5oKHhIwQozdeVoXFHRbPisEcb8Bz+sVt/o5BUBkXkp2n7vF9Y70BU7pdOoUj0 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:3ca0:b0:406:517f:ac75 with SMTP id bg32-20020a05600c3ca000b00406517fac75mr44463wmb.5.1700821289197; Fri, 24 Nov 2023 02:21:29 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:13 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=23616; i=ardb@kernel.org; h=from:subject; bh=Zgn2SnSbGZP7GNYofp/MHUvJWBCGI370dxdw//YiysY=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYm10zxb1E6/y7Bn0vA/vvKBx+GnhvSevE0Tyo2KF0 tc9v1jQUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACby/BjDP71vik/Vos/v9GHw qzzxv7YqhdnstITEnVc+65Zt3rnVkYPhf9GMtuP+/krMby47SFtfP/7FescSHvnuE/e+nVCc8OW pMS8A X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-74-ardb@google.com> Subject: [PATCH v5 33/39] arm64: head: Move early kernel mapping routines into C code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_102131_847970_B18D690F X-CRM114-Status: GOOD ( 24.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The asm version of the kernel mapping code works fine for creating a coarse grained identity map, but for mapping the kernel down to its exact boundaries with the right attributes, it is not suitable. This is why we create a preliminary RWX kernel mapping first, and then rebuild it from scratch later on. So let's reimplement this in C, in a way that will make it unnecessary to create the kernel page tables yet another time in paging_init(). Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/archrandom.h | 2 - arch/arm64/include/asm/scs.h | 32 +--- arch/arm64/kernel/head.S | 52 +----- arch/arm64/kernel/image-vars.h | 19 +++ arch/arm64/kernel/pi/Makefile | 1 + arch/arm64/kernel/pi/idreg-override.c | 22 ++- arch/arm64/kernel/pi/kaslr_early.c | 12 +- arch/arm64/kernel/pi/map_kernel.c | 165 ++++++++++++++++++++ arch/arm64/kernel/pi/map_range.c | 88 +++++++++++ arch/arm64/kernel/pi/patch-scs.c | 16 +- arch/arm64/kernel/pi/pi.h | 14 ++ arch/arm64/kernel/pi/relocate.c | 2 + arch/arm64/kernel/setup.c | 7 - arch/arm64/kernel/vmlinux.lds.S | 4 +- arch/arm64/mm/proc.S | 1 + 15 files changed, 316 insertions(+), 121 deletions(-) diff --git a/arch/arm64/include/asm/archrandom.h b/arch/arm64/include/asm/archrandom.h index ecdb3cfcd0f8..8babfbe31f95 100644 --- a/arch/arm64/include/asm/archrandom.h +++ b/arch/arm64/include/asm/archrandom.h @@ -129,6 +129,4 @@ static inline bool __init __early_cpu_has_rndr(void) return (ftr >> ID_AA64ISAR0_EL1_RNDR_SHIFT) & 0xf; } -u64 kaslr_early_init(void *fdt); - #endif /* _ASM_ARCHRANDOM_H */ diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index eca2ba5a6276..2e010ea76be2 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -33,37 +33,11 @@ #include #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS -static inline bool should_patch_pac_into_scs(void) -{ - u64 reg; - - /* - * We only enable the shadow call stack dynamically if we are running - * on a system that does not implement PAC or BTI. PAC and SCS provide - * roughly the same level of protection, and BTI relies on the PACIASP - * instructions serving as landing pads, preventing us from patching - * those instructions into something else. - */ - reg = read_sysreg_s(SYS_ID_AA64ISAR1_EL1); - if (SYS_FIELD_GET(ID_AA64ISAR1_EL1, APA, reg) | - SYS_FIELD_GET(ID_AA64ISAR1_EL1, API, reg)) - return false; - - reg = read_sysreg_s(SYS_ID_AA64ISAR2_EL1); - if (SYS_FIELD_GET(ID_AA64ISAR2_EL1, APA3, reg)) - return false; - - if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) { - reg = read_sysreg_s(SYS_ID_AA64PFR1_EL1); - if (reg & (0xf << ID_AA64PFR1_EL1_BT_SHIFT)) - return false; - } - return true; -} - static inline void dynamic_scs_init(void) { - if (should_patch_pac_into_scs()) { + extern bool __pi_dynamic_scs_is_enabled; + + if (__pi_dynamic_scs_is_enabled) { pr_info("Enabling dynamic shadow call stack\n"); static_branch_enable(&dynamic_scs_enabled); } diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index aa7766dc64d9..ffacce7b5a02 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -81,7 +81,6 @@ * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob - * x23 __primary_switch() physical misalignment/KASLR offset * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register */ @@ -408,24 +407,6 @@ SYM_FUNC_START_LOCAL(create_idmap) 0: ret x28 SYM_FUNC_END(create_idmap) -SYM_FUNC_START_LOCAL(create_kernel_mapping) - adrp x0, init_pg_dir - mov_q x5, KIMAGE_VADDR // compile time __va(_text) -#ifdef CONFIG_RELOCATABLE - add x5, x5, x23 // add KASLR displacement -#endif - adrp x6, _end // runtime __pa(_end) - adrp x3, _text // runtime __pa(_text) - sub x6, x6, x3 // _end - _text - add x6, x6, x5 // runtime __va(_end) - mov_q x7, SWAPPER_RW_MMUFLAGS - - map_memory x0, x1, x5, x6, x7, x3, (VA_BITS - PGDIR_SHIFT), x10, x11, x12, x13, x14 - - dsb ishst // sync with page table walker - ret -SYM_FUNC_END(create_kernel_mapping) - /* * Initialize CPU registers with task-specific and cpu-specific context. * @@ -752,44 +733,13 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x2, init_idmap_pg_dir bl __enable_mmu - // Clear BSS - adrp x0, __bss_start - mov x1, xzr - adrp x2, init_pg_end - sub x2, x2, x0 - bl __pi_memset - dsb ishst // Make zero page visible to PTW - adrp x1, early_init_stack mov sp, x1 mov x29, xzr mov x0, x20 // pass the full boot status mov x1, x22 // pass the low FDT mapping - bl __pi_init_feature_override // Parse cpu feature overrides - -#ifdef CONFIG_RELOCATABLE - adrp x23, KERNEL_START - and x23, x23, MIN_KIMG_ALIGN - 1 -#ifdef CONFIG_RANDOMIZE_BASE - mov x0, x22 - bl __pi_kaslr_early_init - bic x0, x0, #SZ_2M - 1 - orr x23, x23, x0 // record kernel offset -#endif -#endif - bl create_kernel_mapping + bl __pi_early_map_kernel // Map and relocate the kernel - adrp x1, init_pg_dir - load_ttbr1 x1, x1, x2 -#ifdef CONFIG_RELOCATABLE - mov x0, x23 - bl __pi_relocate_kernel -#endif -#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS - ldr x0, =__eh_frame_start - ldr x1, =__eh_frame_end - bl __pi_scs_patch_vmlinux -#endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) br x8 diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 8d96052079e8..e566b32f9c22 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -45,9 +45,28 @@ PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override); PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); PROVIDE(__pi_id_aa64zfr0_override = id_aa64zfr0_override); PROVIDE(__pi_arm64_sw_feature_override = arm64_sw_feature_override); +PROVIDE(__pi_arm64_use_ng_mappings = arm64_use_ng_mappings); +#ifdef CONFIG_CAVIUM_ERRATUM_27456 +PROVIDE(__pi_cavium_erratum_27456_cpus = cavium_erratum_27456_cpus); +#endif PROVIDE(__pi__ctype = _ctype); PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); +PROVIDE(__pi_init_pg_dir = init_pg_dir); +PROVIDE(__pi_init_pg_end = init_pg_end); + +PROVIDE(__pi__text = _text); +PROVIDE(__pi__stext = _stext); +PROVIDE(__pi__etext = _etext); +PROVIDE(__pi___start_rodata = __start_rodata); +PROVIDE(__pi___inittext_begin = __inittext_begin); +PROVIDE(__pi___inittext_end = __inittext_end); +PROVIDE(__pi___initdata_begin = __initdata_begin); +PROVIDE(__pi___initdata_end = __initdata_end); +PROVIDE(__pi__data = _data); +PROVIDE(__pi___bss_start = __bss_start); +PROVIDE(__pi__end = _end); + #ifdef CONFIG_KVM /* diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index a8b302245f15..8c2f80a46b93 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -39,6 +39,7 @@ $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) obj-y := idreg-override.pi.o \ + map_kernel.pi.o map_range.pi.o \ lib-fdt.pi.o lib-fdt_ro.pi.o obj-$(CONFIG_RELOCATABLE) += relocate.pi.o obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index 1aa59c01ab33..5857e4efad59 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -314,37 +314,35 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) } while (1); } -static __init const u8 *get_bootargs_cmdline(const void *fdt) +static __init const u8 *get_bootargs_cmdline(const void *fdt, int node) { + static char const bootargs[] __initconst = "bootargs"; const u8 *prop; - int node; - node = fdt_path_offset(fdt, "/chosen"); if (node < 0) return NULL; - prop = fdt_getprop(fdt, node, "bootargs", NULL); + prop = fdt_getprop(fdt, node, bootargs, NULL); if (!prop) return NULL; return strlen(prop) ? prop : NULL; } -static __init void parse_cmdline(const void *fdt) +static __init void parse_cmdline(const void *fdt, int chosen) { - const u8 *prop = get_bootargs_cmdline(fdt); + static char const cmdline[] __initconst = CONFIG_CMDLINE; + const u8 *prop = get_bootargs_cmdline(fdt, chosen); if (IS_ENABLED(CONFIG_CMDLINE_FORCE) || !prop) - __parse_cmdline(CONFIG_CMDLINE, true); + __parse_cmdline(cmdline, true); if (!IS_ENABLED(CONFIG_CMDLINE_FORCE) && prop) __parse_cmdline(prop, true); } -/* Keep checkers quiet */ -void init_feature_override(u64 boot_status, const void *fdt); - -asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) +void __init init_feature_override(u64 boot_status, const void *fdt, + int chosen) { struct arm64_ftr_override *override; const struct ftr_set_desc *reg; @@ -360,7 +358,7 @@ asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) __boot_status = boot_status; - parse_cmdline(fdt); + parse_cmdline(fdt, chosen); for (i = 0; i < ARRAY_SIZE(regs); i++) { reg = prel64_to_pointer(®s[i].reg_prel); diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index eeecee7ffd6f..0257b43819db 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -16,17 +16,17 @@ #include #include +#include "pi.h" + extern u16 memstart_offset_seed; -static u64 __init get_kaslr_seed(void *fdt) +static u64 __init get_kaslr_seed(void *fdt, int node) { - static char const chosen_str[] __initconst = "chosen"; static char const seed_str[] __initconst = "kaslr-seed"; - int node, len; fdt64_t *prop; u64 ret; + int len; - node = fdt_path_offset(fdt, chosen_str); if (node < 0) return 0; @@ -39,14 +39,14 @@ static u64 __init get_kaslr_seed(void *fdt) return ret; } -asmlinkage u64 __init kaslr_early_init(void *fdt) +u64 __init kaslr_early_init(void *fdt, int chosen) { u64 seed, range; if (kaslr_disabled_cmdline()) return 0; - seed = get_kaslr_seed(fdt); + seed = get_kaslr_seed(fdt, chosen); if (!seed) { if (!__early_cpu_has_rndr() || !__arm64_rndr((unsigned long *)&seed)) diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c new file mode 100644 index 000000000000..afb6a337fa48 --- /dev/null +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -0,0 +1,165 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Author: Ard Biesheuvel + +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include "pi.h" + +extern const u8 __eh_frame_start[], __eh_frame_end[]; + +extern void idmap_cpu_replace_ttbr1(void *pgdir); + +static void __init map_segment(pgd_t *pg_dir, u64 *pgd, u64 va_offset, + void *start, void *end, pgprot_t prot, + bool may_use_cont, int root_level) +{ + map_range(pgd, ((u64)start + va_offset) & ~PAGE_OFFSET, + ((u64)end + va_offset) & ~PAGE_OFFSET, (u64)start, + prot, root_level, (pte_t *)pg_dir, may_use_cont, 0); +} + +static void __init unmap_segment(pgd_t *pg_dir, u64 va_offset, void *start, + void *end, int root_level) +{ + map_segment(pg_dir, NULL, va_offset, start, end, __pgprot(0), + false, root_level); +} + +static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) +{ + bool enable_scs = IS_ENABLED(CONFIG_UNWIND_PATCH_PAC_INTO_SCS); + bool twopass = IS_ENABLED(CONFIG_RELOCATABLE); + u64 pgdp = (u64)init_pg_dir + PAGE_SIZE; + pgprot_t text_prot = PAGE_KERNEL_ROX; + pgprot_t data_prot = PAGE_KERNEL; + pgprot_t prot; + + /* + * External debuggers may need to write directly to the text mapping to + * install SW breakpoints. Allow this (only) when explicitly requested + * with rodata=off. + */ + if (cpuid_feature_extract_unsigned_field(arm64_sw_feature_override.val, + ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF)) + text_prot = PAGE_KERNEL_EXEC; + + /* + * We only enable the shadow call stack dynamically if we are running + * on a system that does not implement PAC or BTI. PAC and SCS provide + * roughly the same level of protection, and BTI relies on the PACIASP + * instructions serving as landing pads, preventing us from patching + * those instructions into something else. + */ + if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL) && cpu_has_pac()) + enable_scs = false; + + if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL) && cpu_has_bti()) { + enable_scs = false; + + /* + * If we have a CPU that supports BTI and a kernel built for + * BTI then mark the kernel executable text as guarded pages + * now so we don't have to rewrite the page tables later. + */ + text_prot = __pgprot_modify(text_prot, PTE_GP, PTE_GP); + } + + /* Map all code read-write on the first pass if needed */ + twopass |= enable_scs; + prot = twopass ? data_prot : text_prot; + + map_segment(init_pg_dir, &pgdp, va_offset, _stext, _etext, prot, + !twopass, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, __start_rodata, + __inittext_begin, data_prot, false, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, __inittext_begin, + __inittext_end, prot, false, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, __initdata_begin, + __initdata_end, data_prot, false, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, _data, _end, data_prot, + true, root_level); + dsb(ishst); + + idmap_cpu_replace_ttbr1(init_pg_dir); + + if (twopass) { + if (IS_ENABLED(CONFIG_RELOCATABLE)) + relocate_kernel(kaslr_offset); + + if (enable_scs) { + scs_patch(__eh_frame_start + va_offset, + __eh_frame_end - __eh_frame_start); + asm("ic ialluis"); + + dynamic_scs_is_enabled = true; + } + + /* + * Unmap the text region before remapping it, to avoid + * potential TLB conflicts when creating the contiguous + * descriptors. + */ + unmap_segment(init_pg_dir, va_offset, _stext, _etext, + root_level); + dsb(ishst); + isb(); + __tlbi(vmalle1); + isb(); + + /* + * Remap these segments with different permissions + * No new page table allocations should be needed + */ + map_segment(init_pg_dir, NULL, va_offset, _stext, _etext, + text_prot, true, root_level); + map_segment(init_pg_dir, NULL, va_offset, __inittext_begin, + __inittext_end, text_prot, false, root_level); + dsb(ishst); + } +} + +asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) +{ + static char const chosen_str[] __initconst = "/chosen"; + u64 va_base, pa_base = (u64)&_text; + u64 kaslr_offset = pa_base % MIN_KIMG_ALIGN; + int root_level = 4 - CONFIG_PGTABLE_LEVELS; + int chosen; + + /* Clear BSS and the initial page tables */ + memset(__bss_start, 0, (u64)init_pg_end - (u64)__bss_start); + + /* Parse the command line for CPU feature overrides */ + chosen = fdt_path_offset(fdt, chosen_str); + init_feature_override(boot_status, fdt, chosen); + + /* + * The virtual KASLR displacement modulo 2MiB is decided by the + * physical placement of the image, as otherwise, we might not be able + * to create the early kernel mapping using 2 MiB block descriptors. So + * take the low bits of the KASLR offset from the physical address, and + * fill in the high bits from the seed. + */ + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) { + u64 kaslr_seed = kaslr_early_init(fdt, chosen); + + if (kaslr_seed && kaslr_requires_kpti()) + arm64_use_ng_mappings = true; + + kaslr_offset |= kaslr_seed & ~(MIN_KIMG_ALIGN - 1); + } + + va_base = KIMAGE_VADDR + kaslr_offset; + map_kernel(kaslr_offset, va_base - pa_base, root_level); +} diff --git a/arch/arm64/kernel/pi/map_range.c b/arch/arm64/kernel/pi/map_range.c new file mode 100644 index 000000000000..c31feda18f47 --- /dev/null +++ b/arch/arm64/kernel/pi/map_range.c @@ -0,0 +1,88 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Author: Ard Biesheuvel + +#include +#include + +#include +#include +#include + +#include "pi.h" + +/** + * map_range - Map a contiguous range of physical pages into virtual memory + * + * @pte: Address of physical pointer to array of pages to + * allocate page tables from + * @start: Virtual address of the start of the range + * @end: Virtual address of the end of the range (exclusive) + * @pa: Physical address of the start of the range + * @prot: Access permissions of the range + * @level: Translation level for the mapping + * @tbl: The level @level page table to create the mappings in + * @may_use_cont: Whether the use of the contiguous attribute is allowed + * @va_offset: Offset between a physical page and its current mapping + * in the VA space + */ +void __init map_range(u64 *pte, u64 start, u64 end, u64 pa, pgprot_t prot, + int level, pte_t *tbl, bool may_use_cont, u64 va_offset) +{ + u64 cmask = (level == 3) ? CONT_PTE_SIZE - 1 : U64_MAX; + u64 protval = pgprot_val(prot) & ~PTE_TYPE_MASK; + int lshift = (3 - level) * (PAGE_SHIFT - 3); + u64 lmask = (PAGE_SIZE << lshift) - 1; + + start &= PAGE_MASK; + pa &= PAGE_MASK; + + /* Advance tbl to the entry that covers start */ + tbl += (start >> (lshift + PAGE_SHIFT)) % PTRS_PER_PTE; + + /* + * Set the right block/page bits for this level unless we are + * clearing the mapping + */ + if (protval) + protval |= (level < 3) ? PMD_TYPE_SECT : PTE_TYPE_PAGE; + + while (start < end) { + u64 next = min((start | lmask) + 1, PAGE_ALIGN(end)); + + if (level < 3 && (start | next | pa) & lmask) { + /* + * This chunk needs a finer grained mapping. Create a + * table mapping if necessary and recurse. + */ + if (pte_none(*tbl)) { + *tbl = __pte(__phys_to_pte_val(*pte) | + PMD_TYPE_TABLE | PMD_TABLE_UXN); + *pte += PTRS_PER_PTE * sizeof(pte_t); + } + map_range(pte, start, next, pa, prot, level + 1, + (pte_t *)(__pte_to_phys(*tbl) + va_offset), + may_use_cont, va_offset); + } else { + /* + * Start a contiguous range if start and pa are + * suitably aligned + */ + if (((start | pa) & cmask) == 0 && may_use_cont) + protval |= PTE_CONT; + + /* + * Clear the contiguous attribute if the remaining + * range does not cover a contiguous block + */ + if ((end & ~cmask) <= start) + protval &= ~PTE_CONT; + + /* Put down a block or page mapping */ + *tbl = __pte(__phys_to_pte_val(pa) | protval); + } + pa += next - start; + start = next; + tbl++; + } +} diff --git a/arch/arm64/kernel/pi/patch-scs.c b/arch/arm64/kernel/pi/patch-scs.c index c65ef40d1e6b..49d8b40e61bc 100644 --- a/arch/arm64/kernel/pi/patch-scs.c +++ b/arch/arm64/kernel/pi/patch-scs.c @@ -11,6 +11,10 @@ #include +#include "pi.h" + +bool dynamic_scs_is_enabled; + // // This minimal DWARF CFI parser is partially based on the code in // arch/arc/kernel/unwind.c, and on the document below: @@ -46,8 +50,6 @@ #define DW_CFA_GNU_negative_offset_extended 0x2f #define DW_CFA_hi_user 0x3f -extern const u8 __eh_frame_start[], __eh_frame_end[]; - enum { PACIASP = 0xd503233f, AUTIASP = 0xd50323bf, @@ -250,13 +252,3 @@ int scs_patch(const u8 eh_frame[], int size) } return 0; } - -asmlinkage void __init scs_patch_vmlinux(const u8 start[], const u8 end[]) -{ - if (!should_patch_pac_into_scs()) - return; - - scs_patch(start, end - start); - asm("ic ialluis"); - isb(); -} diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h index f455ad385976..8d3d3ba44c27 100644 --- a/arch/arm64/kernel/pi/pi.h +++ b/arch/arm64/kernel/pi/pi.h @@ -2,6 +2,8 @@ // Copyright 2023 Google LLC // Author: Ard Biesheuvel +#include + #define __prel64_initconst __section(".init.rodata.prel64") typedef volatile signed long prel64_t; @@ -12,3 +14,15 @@ static inline void *prel64_to_pointer(const prel64_t *offset) return NULL; return (void *)offset + *offset; } + +extern bool dynamic_scs_is_enabled; + +void init_feature_override(u64 boot_status, const void *fdt, int chosen); +u64 kaslr_early_init(void *fdt, int chosen); +void relocate_kernel(u64 offset); +int scs_patch(const u8 eh_frame[], int size); + +void map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, + int level, pte_t *tbl, bool may_use_cont, u64 va_offset); + +asmlinkage void early_map_kernel(u64 boot_status, void *fdt); diff --git a/arch/arm64/kernel/pi/relocate.c b/arch/arm64/kernel/pi/relocate.c index 1853408ea76b..2407d2696398 100644 --- a/arch/arm64/kernel/pi/relocate.c +++ b/arch/arm64/kernel/pi/relocate.c @@ -7,6 +7,8 @@ #include #include +#include "pi.h" + extern const Elf64_Rela rela_start[], rela_end[]; extern const u64 relr_start[], relr_end[]; diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index c2d6852a4e0c..f75e03b9ad28 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -283,13 +283,6 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) kaslr_init(); - /* - * If know now we are going to need KPTI then use non-global - * mappings from the start, avoiding the cost of rewriting - * everything later. - */ - arm64_use_ng_mappings = kaslr_enabled() && kaslr_requires_kpti(); - early_fixmap_init(); early_ioremap_init(); diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 3afb4223a5e8..755a22d4f840 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -126,9 +126,9 @@ jiffies = jiffies_64; #ifdef CONFIG_UNWIND_TABLES #define UNWIND_DATA_SECTIONS \ .eh_frame : { \ - __eh_frame_start = .; \ + __pi___eh_frame_start = .; \ *(.eh_frame) \ - __eh_frame_end = .; \ + __pi___eh_frame_end = .; \ } #else #define UNWIND_DATA_SECTIONS diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index f66c37a1610e..7c1bdaf25408 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -195,6 +195,7 @@ SYM_TYPED_FUNC_START(idmap_cpu_replace_ttbr1) ret SYM_FUNC_END(idmap_cpu_replace_ttbr1) +SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .popsection #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 From patchwork Fri Nov 24 10:19:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467460 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ADC88C636CB for ; Fri, 24 Nov 2023 10:23:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=MsAtw/0FNe0B374Z2l266x1NgFpbR2HqIPtCUqpzDMc=; b=IsmcBy+c4L9pClnajfDb9BqXUs Jd0By8MtWBxi7vDvwsJ888hnTbMqtkmH5M0MnSUeFKkcKCAYGgvyOexr4ImBdyu625Ac+MeEcsQBv gJEAcc3wCTfmBxytPgFq4g9F8q4lttAjwWPX2atQVCTZUW7yaLChyYSOz2vX6LWSuiNXNA5BX/GZh wOeSUCAsE3g2Wb6qYKs/Vln/zKR0gRZDXMNBgojetoXvG3p/FZ61OcW4pPF37O4QMtzSpKs3jBtNm Lvwq0W98hN+DCcVI7355ASHGbHPp1Ad9fgC81y1/ZjNhMe2qcFYIAM9Xi2WGh6keaqxKfdYaCxnCB T4lppEhA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TL4-006pwL-2N; Fri, 24 Nov 2023 10:23:10 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TKA-006pc8-2H for linux-arm-kernel@bombadil.infradead.org; Fri, 24 Nov 2023 10:22:14 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=via/947AL7nxdvgSE2nro4c+kFND3Teh9zjacOXE+kk=; b=JDYi09Cc7sVEcmWViHdZN61XO8 Pzc7499PMkMRJmMHPf1m4TTvq5usIpeQLpaZGdEQNmnj3ZSZHFmfHIzwtFzFlGtR3GXmCEPCNcrWr ppmjmmKtkvahep7zwnRH59ABHjxmRrB0uJOzPXcTtwWqIA2vB6QTHIruhJUFN+oXCPI1isHKYRkmC 6HHtRjsZm6crHr/S3Cpe4UL+Az21WF1w8TPIvCY5ak+q1QpKkBX0/3zNpG1SIMb7rWHgfJ1Pg64gr wudezvAYewde9mFTKDcKt9F3vyzvaauOghoYP4zF3TFBUAqyNL7e14+yr56k1hR8nBBelRC3Y0XCg pJ3AL8gA==; Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJu-00DqX5-25 for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:22:11 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-40b2977d6baso8984925e9.2 for ; Fri, 24 Nov 2023 02:21:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821291; x=1701426091; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=via/947AL7nxdvgSE2nro4c+kFND3Teh9zjacOXE+kk=; b=TCc7Sn6pZcR2RyO8q+t7w+h8fLbG4/aH/ryoTSY77bxKYwyFiauXwBqAJZufS26ngE txjqAdzMcpHMijKXZolOWP/gJklIQMpZ6kXviLJn4iRDi6FRkTYKmf7Fpnv+lYp7SjEB 2AyyrIW/9cXFxNeRgWWIvzoRzyDhvGI2hphpXXvwfN6oqmoIgvGZzR39x6SKdqOoaCYa DN1CQaooF9PYaQky+q4qMhe04cb/NjOZoNL3euFvvk7GkPAIH+aGbG6WaKwDCpKppUe7 /AQmUzC409EYKRozV+O1BKOfB7/vzrVMpYQ9nTBQL46LcgJmtptWbh/K0pAvlj7DkCgm n/jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821291; x=1701426091; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=via/947AL7nxdvgSE2nro4c+kFND3Teh9zjacOXE+kk=; b=RJSTBYDzSKmc1IEOl29zCIxLOVyCCb5eisC/vkjZB/tp577PUgOQ9+RNSiaBQCQ98V y7Djaiu6V7NS3oU9ucerb2XnLLN56R5Hnqjw8mZ5bsPtuS4+lTfPfXJVMcoHemVZw5rp BjbaIcHXKQ0LkigEBjh9iwSthN+Sxzga82oH6R2txykFA7F1cEy9GLtLBQA8vDwwRa7r JUUk5iDqebkNFd7PLMz5EUaSKVhgSuq7MtSdigF1P3bnhwmo6CY9FSwjoH5gEp6Hkngd 7NX/HuaR3nYL+eeaO1C2/7L4vR6WzcFWF6udplkw0ZnI9vseuNoCQr6pwyo5H6N1JQJT m5FA== X-Gm-Message-State: AOJu0Yzgo+CmMw9KU+MjtEIoH2XbHFWsq5chg64DjaqSjrCweFFXnamh vIi/c75D/Bu3OD3KDePwMo2Fc5IS668GkgLTLUUEmahvpu77XLgIu9vQ7rirKK2UWmotW4V2PZp FDRr235ftM6wmQiR7DreIRdAPRhI+MesUAE5gogNNjYbD4t8MXGzDFkJ+VXr/6VAV/Aref8BYiO M= X-Google-Smtp-Source: AGHT+IHm36msVnMBXl33QHj8GorzwrPvPaR0u3k5ghjYFMKvMpUioUpD/47euE4C+Y/rm+z6ZkvzJLdi X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:b60a:0:b0:332:ddd8:859b with SMTP id f10-20020adfb60a000000b00332ddd8859bmr42532wre.2.1700821291564; Fri, 24 Nov 2023 02:21:31 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:14 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=7014; i=ardb@kernel.org; h=from:subject; bh=V8rU2zh2WeBn4WszMWjet22dmrtBYY4KlyDB/jHLvnc=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYt21RdHXJa0TzMsagnt+vJ7ytkBZhn9qQwBbWNrjz fGzup51lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgInsM2NkOK7wIkzk04KQS2eX n3fkUzC1+bgpsnWGjPaqy8ecT+l6CzAydBpVL7xgtUvqOc/bCSvqk3etU3WtSP8raVcauPWtw11 dFgA= X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-75-ardb@google.com> Subject: [PATCH v5 34/39] arm64: mm: Use 48-bit virtual addressing for the permanent ID map From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_102159_020409_102CDC03 X-CRM114-Status: GOOD ( 24.12 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Even though we support loading kernels anywhere in 48-bit addressable physical memory, we create the ID maps based on the number of levels that we happened to configure for the kernel VA and user VA spaces. The reason for this is that the PGD/PUD/PMD based classification of translation levels, along with the associated folding when the number of levels is less than 5, does not permit creating a page table hierarchy of a set number of levels. This means that, for instance, on 39-bit VA kernels we need to configure an additional level above PGD level on the fly, and 36-bit VA kernels still only support 47-bit virtual addressing with this trick applied. Now that we have a separate helper to populate page table hierarchies that does not define the levels in terms of PUDS/PMDS/etc at all, let's reuse it to create the permanent ID map with a fixed VA size of 48 bits. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 3 ++ arch/arm64/kernel/head.S | 5 +++ arch/arm64/kvm/mmu.c | 15 +++------ arch/arm64/mm/mmu.c | 32 +++++++++++--------- arch/arm64/mm/proc.S | 9 ++---- 5 files changed, 32 insertions(+), 32 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 0631604995ee..742a4b2778f7 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -35,6 +35,9 @@ #define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS) #endif +#define IDMAP_VA_BITS 48 +#define IDMAP_LEVELS ARM64_HW_PGTABLE_LEVELS(IDMAP_VA_BITS) +#define IDMAP_ROOT_LEVEL (4 - IDMAP_LEVELS) /* * A relocatable kernel may execute from an address that differs from the one at diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index ffacce7b5a02..a1c29d64e875 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -729,6 +729,11 @@ SYM_FUNC_START_LOCAL(__no_granule_support) SYM_FUNC_END(__no_granule_support) SYM_FUNC_START_LOCAL(__primary_switch) + mrs x1, tcr_el1 + mov x2, #64 - VA_BITS + tcr_set_t0sz x1, x2 + msr tcr_el1, x1 + adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index d87c8fcc4c24..234b03e18ff7 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1873,16 +1873,9 @@ int __init kvm_mmu_init(u32 *hyp_va_bits) BUG_ON((hyp_idmap_start ^ (hyp_idmap_end - 1)) & PAGE_MASK); /* - * The ID map may be configured to use an extended virtual address - * range. This is only the case if system RAM is out of range for the - * currently configured page size and VA_BITS_MIN, in which case we will - * also need the extended virtual range for the HYP ID map, or we won't - * be able to enable the EL2 MMU. - * - * However, in some cases the ID map may be configured for fewer than - * the number of VA bits used by the regular kernel stage 1. This - * happens when VA_BITS=52 and the kernel image is placed in PA space - * below 48 bits. + * The ID map is always configured for 48 bits of translation, which + * may be fewer than the number of VA bits used by the regular kernel + * stage 1, when VA_BITS=52. * * At EL2, there is only one TTBR register, and we can't switch between * translation tables *and* update TCR_EL2.T0SZ at the same time. Bottom @@ -1893,7 +1886,7 @@ int __init kvm_mmu_init(u32 *hyp_va_bits) * 1 VA bits to assure that the hypervisor can both ID map its code page * and map any kernel memory. */ - idmap_bits = 64 - ((idmap_t0sz & TCR_T0SZ_MASK) >> TCR_T0SZ_OFFSET); + idmap_bits = IDMAP_VA_BITS; kernel_bits = vabits_actual; *hyp_va_bits = max(idmap_bits, kernel_bits); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 03c73e9197ac..94847dbd31cd 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -754,22 +754,21 @@ static void __init map_kernel(pgd_t *pgdp) kasan_copy_shadow(pgdp); } +void __pi_map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, + int level, pte_t *tbl, bool may_use_cont, u64 va_offset); + +static u8 idmap_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init, + kpti_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init; + static void __init create_idmap(void) { u64 start = __pa_symbol(__idmap_text_start); - u64 size = __pa_symbol(__idmap_text_end) - start; - pgd_t *pgd = idmap_pg_dir; - u64 pgd_phys; - - /* check if we need an additional level of translation */ - if (VA_BITS < 48 && idmap_t0sz < (64 - VA_BITS_MIN)) { - pgd_phys = early_pgtable_alloc(PAGE_SHIFT); - set_pgd(&idmap_pg_dir[start >> VA_BITS], - __pgd(pgd_phys | P4D_TYPE_TABLE)); - pgd = __va(pgd_phys); - } - __create_pgd_mapping(pgd, start, start, size, PAGE_KERNEL_ROX, - early_pgtable_alloc, 0); + u64 end = __pa_symbol(__idmap_text_end); + u64 ptep = __pa_symbol(idmap_ptes); + + __pi_map_range(&ptep, start, end, start, PAGE_KERNEL_ROX, + IDMAP_ROOT_LEVEL, (pte_t *)idmap_pg_dir, false, + __phys_to_virt(ptep) - ptep); if (IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0)) { extern u32 __idmap_kpti_flag; @@ -779,8 +778,10 @@ static void __init create_idmap(void) * The KPTI G-to-nG conversion code needs a read-write mapping * of its synchronization flag in the ID map. */ - __create_pgd_mapping(pgd, pa, pa, sizeof(u32), PAGE_KERNEL, - early_pgtable_alloc, 0); + ptep = __pa_symbol(kpti_ptes); + __pi_map_range(&ptep, pa, pa + sizeof(u32), pa, PAGE_KERNEL, + IDMAP_ROOT_LEVEL, (pte_t *)idmap_pg_dir, false, + __phys_to_virt(ptep) - ptep); } } @@ -805,6 +806,7 @@ void __init paging_init(void) memblock_allow_resize(); create_idmap(); + idmap_t0sz = TCR_T0SZ(IDMAP_VA_BITS); } #ifdef CONFIG_MEMORY_HOTPLUG diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 7c1bdaf25408..47ede52bb900 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -421,9 +421,9 @@ SYM_FUNC_START(__cpu_setup) mair .req x17 tcr .req x16 mov_q mair, MAIR_EL1_SET - mov_q tcr, TCR_TxSZ(VA_BITS) | TCR_CACHE_FLAGS | TCR_SMP_FLAGS | \ - TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ - TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS + mov_q tcr, TCR_T0SZ(IDMAP_VA_BITS) | TCR_T1SZ(VA_BITS) | TCR_CACHE_FLAGS | \ + TCR_SMP_FLAGS | TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ + TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS tcr_clear_errata_bits tcr, x9, x5 @@ -431,10 +431,7 @@ SYM_FUNC_START(__cpu_setup) sub x9, xzr, x0 add x9, x9, #64 tcr_set_t1sz tcr, x9 -#else - idmap_get_t0sz x9 #endif - tcr_set_t0sz tcr, x9 /* * Set the IPS bits in TCR_EL1. From patchwork Fri Nov 24 10:19:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 973F2C61DF4 for ; Fri, 24 Nov 2023 10:23:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=xwHkk4ksK65UUzQ8XTUFpehDJFUNJHZXoxeYCghyQnM=; b=zunnNsKEWN8FJysVl++3f+Icn7 Zpdyws/h7FCo3+Jz5YhuRst656eehE3vzrf9+uTbAIqdXkEdH9Gwq8SNpBNSPF5tIGSVYCKErDyWe 7razxx8G57iZZV4xI/uvMPjDUfNi/AI2iGKpsUs/zGg3KjEiq3G41EN4/TGme5NFZ03TvXBSHtvFE lir3klWTkulYbcopzzCgYpGV62r1tY2pdSv6VTQnprCyLhK6Ea0S5q7IwyOx5qbZ/unt3tt4H3WhS hR3/Hvd6GFdANA/Sm7edAZJzLRqup0lcKLA33iIa1/E1iXGTI94rVK43weWBoBzAFuslflmUXH9p/ p8Ay7MRw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TKz-006psu-25; Fri, 24 Nov 2023 10:23:05 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJY-006pFN-3B for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:49 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-332e11a22a0so879152f8f.2 for ; Fri, 24 Nov 2023 02:21:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821294; x=1701426094; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mJgEdoeKR1YMff1vw9Z8TUCFCAyrBcTVbGoDtrpkcs8=; b=pd6H8LJpmUVpdYVFnuGKl7JmtZXOze6G63Tr+0hd6blhaaLaACRcdym5sjfpRfnO9H oqupgB/tXIIHCiOTjmCCyDbHWgCpCBk9wR6GyW42sEEWRxCs+HvKjsj7vhfS83+5eKza PXIgIA0UKU8PP5rINLTUI09woisOxqCmUrsCd8OO29htonOD6GfpkvGy9ckv5widyOAN IGGMy88T1tW0Qih1OScw9/qMN7Ce6fClDb5qiDCXakgFhBlHPm73hHS3brPZ9ymeCvvG YHSIJhqNWg85RChxg7tKIbXHbl32pqU7ASyI5mjIvfTUi6tS7ieGRmPutN01VCEwz3yE AW5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821294; x=1701426094; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mJgEdoeKR1YMff1vw9Z8TUCFCAyrBcTVbGoDtrpkcs8=; b=GBSs6Gs2IoxVSMCXioPinCN8KnHrog2JDFIjITXvKBZUYm1IuswNsI5v1jxp2PMjTH V7PTvU0AgwwfjZizlK/cz4Kna8Jp9POFoKxINVyCkfDSMyIQ2kkCGC+Z0DiqBg9oPzqK 5aFMyR61N0OWpLnMd0ndsKuB72VlE5mZf0DEtBrRknl1HAmvkHLWJF/5wm6tjv4AgIKa Qh+ZMd6m/zNO9HjjS9IIdZktR868vFs3ws5d0ZNm3fzaRH3wyKGn/h+RZAc2PNx6HEVe 7FXvBqnbuiVWnf2ie2R3G+jbrUGAtstUN3gJXGpKhhBMcCk8elc1vN2oRwEf4LhmnNUp 4KsA== X-Gm-Message-State: AOJu0YzmmccnTA1JyDyyuvow01vsLF7DRH4KQWTa99JR0jWRkoxg1pM6 jMtdM4C9TZ1YaMTUTIia+YSDOt7wCOvlmm3rqmHSSrnlSuFuWFnFyU9ZRm5UP26nNAS12IaCoTe YoG4krGIXf1N/JQWdC0TAsFzoieSY7k8EpHpmBKxXZytud4WLtLhBP59pI7G2T0YPGQq+zZW6DU s= X-Google-Smtp-Source: AGHT+IF7i63j4DMd/T0VsobpBh4NmkpDg/L++ezYBLCukmzuDXbgKh3wJFKmzM5MHHZqdSh9zc6oRZit X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:dd81:0:b0:32f:b3eb:c3d8 with SMTP id x1-20020adfdd81000000b0032fb3ebc3d8mr34650wrl.5.1700821293684; Fri, 24 Nov 2023 02:21:33 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:15 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5172; i=ardb@kernel.org; h=from:subject; bh=xUwSPPT81eaQTdUSndo82jx0GrRomul61QO77P+qcOg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYn2wV+8rUefCBt4W3bOs/uVqbEVfhMMC+HZ9vL1zc 3Xb2dMdpSwMYhwMsmKKLAKz/77beXqiVK3zLFmYOaxMIEMYuDgFYCLLJBj+GXxRcjq5TPOEyM+Q ezX/bkud/F2xVrh8gpFz7v7fk28v9GZkuPRCRUbj4JOKrasVBNweXdE2eNDDYiJ1p3+eVnXuUt8 dDAA= X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-76-ardb@google.com> Subject: [PATCH v5 35/39] arm64: pgtable: Decouple PGDIR size macros from PGD/PUD/PMD levels From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022137_055681_E4A31B23 X-CRM114-Status: GOOD ( 18.56 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The mapping from PGD/PUD/PMD to levels and shifts is very confusing, given that, due to folding, the shifts may be equal for different levels, if the macros are even #define'd to begin with. In a subsequent patch, we will modify the ID mapping code to decouple the number of levels from the kernel's view of how these types are folded, so prepare for this by reformulating the macros without the use of these types. Instead, use SWAPPER_BLOCK_SHIFT as the base quantity, and derive it from either PAGE_SHIFT or PMD_SHIFT, which -if defined at all- are defined unambiguously for a given page size, regardless of the number of configured levels. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 65 ++++++-------------- 1 file changed, 19 insertions(+), 46 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 742a4b2778f7..f1fc98a233d5 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -13,27 +13,22 @@ #include /* - * The linear mapping and the start of memory are both 2M aligned (per - * the arm64 booting.txt requirements). Hence we can use section mapping - * with 4K (section size = 2M) but not with 16K (section size = 32M) or - * 64K (section size = 512M). + * The physical and virtual addresses of the start of the kernel image are + * equal modulo 2 MiB (per the arm64 booting.txt requirements). Hence we can + * use section mapping with 4K (section size = 2M) but not with 16K (section + * size = 32M) or 64K (section size = 512M). */ - -/* - * The idmap and swapper page tables need some space reserved in the kernel - * image. Both require pgd, pud (4 levels only) and pmd tables to (section) - * map the kernel. With the 64K page configuration, swapper and idmap need to - * map to pte level. The swapper also maps the FDT (see __create_page_tables - * for more information). Note that the number of ID map translation levels - * could be increased on the fly if system RAM is out of reach for the default - * VA range, so pages required to map highest possible PA are reserved in all - * cases. - */ -#ifdef CONFIG_ARM64_4K_PAGES -#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - 1) +#if defined(PMD_SIZE) && PMD_SIZE <= MIN_KIMG_ALIGN +#define SWAPPER_BLOCK_SHIFT PMD_SHIFT +#define SWAPPER_SKIP_LEVEL 1 #else -#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS) +#define SWAPPER_BLOCK_SHIFT PAGE_SHIFT +#define SWAPPER_SKIP_LEVEL 0 #endif +#define SWAPPER_BLOCK_SIZE (UL(1) << SWAPPER_BLOCK_SHIFT) +#define SWAPPER_TABLE_SHIFT (SWAPPER_BLOCK_SHIFT + PAGE_SHIFT - 3) + +#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - SWAPPER_SKIP_LEVEL) #define IDMAP_VA_BITS 48 #define IDMAP_LEVELS ARM64_HW_PGTABLE_LEVELS(IDMAP_VA_BITS) @@ -53,24 +48,13 @@ #define EARLY_ENTRIES(vstart, vend, shift, add) \ (SPAN_NR_ENTRIES(vstart, vend, shift) + (add)) -#define EARLY_PGDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, PGDIR_SHIFT, add)) - -#if SWAPPER_PGTABLE_LEVELS > 3 -#define EARLY_PUDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, PUD_SHIFT, add)) -#else -#define EARLY_PUDS(vstart, vend, add) (0) -#endif +#define EARLY_LEVEL(lvl, vstart, vend, add) \ + (SWAPPER_PGTABLE_LEVELS > lvl ? EARLY_ENTRIES(vstart, vend, SWAPPER_BLOCK_SHIFT + lvl * (PAGE_SHIFT - 3), add) : 0) -#if SWAPPER_PGTABLE_LEVELS > 2 -#define EARLY_PMDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, SWAPPER_TABLE_SHIFT, add)) -#else -#define EARLY_PMDS(vstart, vend, add) (0) -#endif - -#define EARLY_PAGES(vstart, vend, add) ( 1 /* PGDIR page */ \ - + EARLY_PGDS((vstart), (vend), add) /* each PGDIR needs a next level page table */ \ - + EARLY_PUDS((vstart), (vend), add) /* each PUD needs a next level page table */ \ - + EARLY_PMDS((vstart), (vend), add)) /* each PMD needs a next level page table */ +#define EARLY_PAGES(vstart, vend, add) (1 /* PGDIR page */ \ + + EARLY_LEVEL(3, (vstart), (vend), add) /* each entry needs a next level page table */ \ + + EARLY_LEVEL(2, (vstart), (vend), add) /* each entry needs a next level page table */ \ + + EARLY_LEVEL(1, (vstart), (vend), add))/* each entry needs a next level page table */ #define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE) + EARLY_SEGMENT_EXTRA_PAGES)) /* the initial ID map may need two extra pages if it needs to be extended */ @@ -81,17 +65,6 @@ #endif #define INIT_IDMAP_DIR_PAGES EARLY_PAGES(KIMAGE_VADDR, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE, 1) -/* Initial memory map size */ -#ifdef CONFIG_ARM64_4K_PAGES -#define SWAPPER_BLOCK_SHIFT PMD_SHIFT -#define SWAPPER_BLOCK_SIZE PMD_SIZE -#define SWAPPER_TABLE_SHIFT PUD_SHIFT -#else -#define SWAPPER_BLOCK_SHIFT PAGE_SHIFT -#define SWAPPER_BLOCK_SIZE PAGE_SIZE -#define SWAPPER_TABLE_SHIFT PMD_SHIFT -#endif - /* The number of segments in the kernel image (text, rodata, inittext, initdata, data+bss) */ #define KERNEL_SEGMENT_COUNT 5 From patchwork Fri Nov 24 10:19:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 63105C61DF7 for ; Fri, 24 Nov 2023 10:23:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=bjhK1u6P9wk48Mth6AzsEuPTYPJ7rIqWvdAKyfxSpUQ=; b=EcEXvvDF3jpH8IAHJ5fZB7iwQB A98u/6PmJwdoGTk5cVDhi9PqkFhX0gNzlRN8RKH+VbkzLn7AXLmzwCkerZOQZheldkiWGeRNlin9g gu1d/Hy2ew/hFeuy27ahad7+9e3vWVdCBJfAek0lBDpd8GekOQ2KRRgcikblVAMNr/9VYRwVrG75O uxqqp92CbaZ3hcxg5m1Tg4x7KskAleezb/9CPECibj+7v1IBzhRogdavhRodGk+oOHB2ysD23rsj5 IeWDbREPDP9aGb02EtKVana9RKgeOnk34Ve8HNa5WoUcXXNmPwxuNTL+5y2asfKHzzVx1AxuK8PL7 z3RU7oDw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TL0-006ptN-0r; Fri, 24 Nov 2023 10:23:06 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJZ-006pGe-2G for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:55 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-40b23aef363so9417705e9.1 for ; Fri, 24 Nov 2023 02:21:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821295; x=1701426095; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=98r3ZY3QHxuIn+hkbLnEXkvd8+BzxeZxN3s0dm9tu/c=; b=Qe5qIqatnDeMH6eDusUX3/msPqMPoSoays0ANTmoy/Ogp+BlDSPent2oidWRmlkOgM yZnmrhxdbRlBsyEaUCNBcae3FW4b1zsTTrpupehk9Jvgrpki21Z+SvJo5Gv2XhAR4ZTI uTUBAW6fEbmkBfiGKoxT3ov3Yy8QESNOUfqHA1KdYqd+8l5yEFzNFptS+oC5kJSjOkBC dEn//cOAmiY6nDrMgSHvoW5qFeuaHyyavxFVctyQgei1ZKCVM1DEvkblqMQrzKmf3Ped dSZZM8oT2yGnGt7UEZQMa2IglwAHP7wtL50Sy2DcI2g2OOB3NLN32kXlEoh20O7FoG+d 2rYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821295; x=1701426095; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=98r3ZY3QHxuIn+hkbLnEXkvd8+BzxeZxN3s0dm9tu/c=; b=hDjr1Tgkndu5y/XIyxGnY8ZfVtppxCVmqjO3eR/DVIDgJkbD9gp4TIAAEM8Z2oJeQW gJLeMuCd9TRbOI+IThmLaqi3Zm39JL7XT0TdHWoA97UrJdAT4qwx0hhdNlIrsjGeIZQX WJyto9RAdXrlZGIDUzRA4bD+7jZPnPur/fGMSUiOwwHHiHtiuM4Z/OFY2f3zItXRkaor tCy822L/BsLt3WHGG/qrqEfff9jPIQQA1bROUd8bTD6SWfXBlUtryuN5+fmP6qQ5PDK1 wia7+HtmSZVD1WoWbygbfa4rRNnna9hh3OG66PzWpAsXxBE+mZjAbJmPiE2xBbcfGK85 K1iA== X-Gm-Message-State: AOJu0YzvZFtzl8dhn6Gkpj32jKTdiIqUs0Kedf4pBBt0wkyK1+ZOI0vS ca0ZO/Y9OXGpxLRnEhUX7dBO4R34uQ6HCRW+kcZlTGN0y9P+RmWm0P+rME9wl1bQ2m2Cz1kZ1mh UIXaSOtFUaaBKZ+3BesGhMEAmups+kQuVgPn/qhgROlUXA/+4PGMqSqtL0BWAiu4c3r9065sCgf U= X-Google-Smtp-Source: AGHT+IG1UAhFkaiYmXCCHZgw9G2raHsFhd73bCyGcpLsIn9NkMJb9vLFjpcGbCSqAWOPNOzbuQmXlTNy X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:b60d:0:b0:32f:7c55:e846 with SMTP id f13-20020adfb60d000000b0032f7c55e846mr34011wre.0.1700821295710; Fri, 24 Nov 2023 02:21:35 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:16 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=24477; i=ardb@kernel.org; h=from:subject; bh=+BFHWOp94m02EgZYBdMP/5Cs8ggl5nj/48KaEDbhTSk=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYuP6L8eq1n29f1tr/mYlrvw4LYn7XlcdKhTvb/0d9 5ZrO9PPjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRjp0MfziunSnbXdNx+k3J g/s7YibKssScXJq7x2DnbzPpy8ea/nIz/GZ/X3F73RIZD4bH8i2pmxb7GYULP5mtoRelaHX9rth lby4A X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-77-ardb@google.com> Subject: [PATCH v5 36/39] arm64: kernel: Create initial ID map from C code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022137_778637_07C1F590 X-CRM114-Status: GOOD ( 32.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The asm code that creates the initial ID map is rather intricate and hard to follow. This is problematic because it makes adding support for things like LPA2 or WXN more difficult than necessary. Also, it is parameterized like the rest of the MM code to run with a configurable number of levels, which is rather pointless, given that all AArch64 CPUs implement support for 48-bit virtual addressing, and that many systems exist with DRAM located outside of the 39-bit addressable range, which is the only smaller VA size that is widely used, and we need additional tricks to make things work in that combination. So let's bite the bullet, and rip out all the asm macros, and fiddly code, and replace it with a C implementation based on the newly added routines for creating the early kernel VA mappings. And while at it, create the initial ID map based on 48-bit virtual addressing as well, regardless of the number of configured levels for the kernel proper. Note that this code may execute with the MMU and caches disabled, and is therefore not permitted to make unaligned accesses. This shouldn't generally happen in any case for the algorithm as implemented, but to be sure, let's pass -mstrict-align to the compiler just in case. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 14 - arch/arm64/include/asm/kernel-pgtable.h | 50 ++-- arch/arm64/include/asm/mmu_context.h | 6 +- arch/arm64/kernel/head.S | 267 ++------------------ arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/Makefile | 3 + arch/arm64/kernel/pi/map_kernel.c | 18 ++ arch/arm64/kernel/pi/map_range.c | 12 + arch/arm64/kernel/pi/pi.h | 4 + arch/arm64/mm/mmu.c | 5 - arch/arm64/mm/proc.S | 3 +- 11 files changed, 88 insertions(+), 295 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 376a980f2bad..beb53bbd8c19 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -345,20 +345,6 @@ alternative_cb_end bfi \valreg, \t1sz, #TCR_T1SZ_OFFSET, #TCR_TxSZ_WIDTH .endm -/* - * idmap_get_t0sz - get the T0SZ value needed to cover the ID map - * - * Calculate the maximum allowed value for TCR_EL1.T0SZ so that the - * entire ID map region can be mapped. As T0SZ == (64 - #bits used), - * this number conveniently equals the number of leading zeroes in - * the physical address of _end. - */ - .macro idmap_get_t0sz, reg - adrp \reg, _end - orr \reg, \reg, #(1 << VA_BITS_MIN) - 1 - clz \reg, \reg - .endm - /* * tcr_compute_pa_size - set TCR.(I)PS to the highest supported * ID_AA64MMFR0_EL1.PARange value diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index f1fc98a233d5..bf05a77873a4 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -29,6 +29,7 @@ #define SWAPPER_TABLE_SHIFT (SWAPPER_BLOCK_SHIFT + PAGE_SHIFT - 3) #define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - SWAPPER_SKIP_LEVEL) +#define INIT_IDMAP_PGTABLE_LEVELS (IDMAP_LEVELS - SWAPPER_SKIP_LEVEL) #define IDMAP_VA_BITS 48 #define IDMAP_LEVELS ARM64_HW_PGTABLE_LEVELS(IDMAP_VA_BITS) @@ -48,44 +49,39 @@ #define EARLY_ENTRIES(vstart, vend, shift, add) \ (SPAN_NR_ENTRIES(vstart, vend, shift) + (add)) -#define EARLY_LEVEL(lvl, vstart, vend, add) \ - (SWAPPER_PGTABLE_LEVELS > lvl ? EARLY_ENTRIES(vstart, vend, SWAPPER_BLOCK_SHIFT + lvl * (PAGE_SHIFT - 3), add) : 0) +#define EARLY_LEVEL(lvl, lvls, vstart, vend, add) \ + (lvls > lvl ? EARLY_ENTRIES(vstart, vend, SWAPPER_BLOCK_SHIFT + lvl * (PAGE_SHIFT - 3), add) : 0) -#define EARLY_PAGES(vstart, vend, add) (1 /* PGDIR page */ \ - + EARLY_LEVEL(3, (vstart), (vend), add) /* each entry needs a next level page table */ \ - + EARLY_LEVEL(2, (vstart), (vend), add) /* each entry needs a next level page table */ \ - + EARLY_LEVEL(1, (vstart), (vend), add))/* each entry needs a next level page table */ -#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE) + EARLY_SEGMENT_EXTRA_PAGES)) +#define EARLY_PAGES(lvls, vstart, vend, add) (1 /* PGDIR page */ \ + + EARLY_LEVEL(3, (lvls), (vstart), (vend), add) /* each entry needs a next level page table */ \ + + EARLY_LEVEL(2, (lvls), (vstart), (vend), add) /* each entry needs a next level page table */ \ + + EARLY_LEVEL(1, (lvls), (vstart), (vend), add))/* each entry needs a next level page table */ +#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(SWAPPER_PGTABLE_LEVELS, KIMAGE_VADDR, _end, EXTRA_PAGE) \ + + EARLY_SEGMENT_EXTRA_PAGES)) -/* the initial ID map may need two extra pages if it needs to be extended */ -#if VA_BITS < 48 -#define INIT_IDMAP_DIR_SIZE ((INIT_IDMAP_DIR_PAGES + 2) * PAGE_SIZE) -#else -#define INIT_IDMAP_DIR_SIZE (INIT_IDMAP_DIR_PAGES * PAGE_SIZE) -#endif -#define INIT_IDMAP_DIR_PAGES EARLY_PAGES(KIMAGE_VADDR, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE, 1) +#define INIT_IDMAP_DIR_PAGES (EARLY_PAGES(INIT_IDMAP_PGTABLE_LEVELS, KIMAGE_VADDR, _end, 1)) +#define INIT_IDMAP_DIR_SIZE ((INIT_IDMAP_DIR_PAGES + EARLY_IDMAP_EXTRA_PAGES) * PAGE_SIZE) + +#define INIT_IDMAP_FDT_PAGES (EARLY_PAGES(INIT_IDMAP_PGTABLE_LEVELS, 0UL, UL(MAX_FDT_SIZE), 1) - 1) +#define INIT_IDMAP_FDT_SIZE ((INIT_IDMAP_FDT_PAGES + EARLY_IDMAP_EXTRA_FDT_PAGES) * PAGE_SIZE) /* The number of segments in the kernel image (text, rodata, inittext, initdata, data+bss) */ #define KERNEL_SEGMENT_COUNT 5 #if SWAPPER_BLOCK_SIZE > SEGMENT_ALIGN #define EARLY_SEGMENT_EXTRA_PAGES (KERNEL_SEGMENT_COUNT + 1) -#else -#define EARLY_SEGMENT_EXTRA_PAGES 0 -#endif - /* - * Initial memory map attributes. + * The initial ID map consists of the kernel image, mapped as two separate + * segments, and may appear misaligned wrt the swapper block size. This means + * we need 3 additional pages. The DT could straddle a swapper block boundary, + * so it may need 2. */ -#define SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED | PTE_UXN) -#define SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S | PTE_UXN) - -#ifdef CONFIG_ARM64_4K_PAGES -#define SWAPPER_RW_MMUFLAGS (PMD_ATTRINDX(MT_NORMAL) | SWAPPER_PMD_FLAGS | PTE_WRITE) -#define SWAPPER_RX_MMUFLAGS (SWAPPER_RW_MMUFLAGS | PMD_SECT_RDONLY) +#define EARLY_IDMAP_EXTRA_PAGES 3 +#define EARLY_IDMAP_EXTRA_FDT_PAGES 2 #else -#define SWAPPER_RW_MMUFLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS | PTE_WRITE) -#define SWAPPER_RX_MMUFLAGS (SWAPPER_RW_MMUFLAGS | PTE_RDONLY) +#define EARLY_SEGMENT_EXTRA_PAGES 0 +#define EARLY_IDMAP_EXTRA_PAGES 0 +#define EARLY_IDMAP_EXTRA_FDT_PAGES 0 #endif #endif /* __ASM_KERNEL_PGTABLE_H */ diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 9ce4200508b1..3c487948b94e 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -61,11 +61,9 @@ static inline void cpu_switch_mm(pgd_t *pgd, struct mm_struct *mm) } /* - * TCR.T0SZ value to use when the ID map is active. Usually equals - * TCR_T0SZ(VA_BITS), unless system RAM is positioned very high in - * physical memory, in which case it will be smaller. + * TCR.T0SZ value to use when the ID map is active. */ -extern int idmap_t0sz; +#define idmap_t0sz TCR_T0SZ(IDMAP_VA_BITS) /* * Ensure TCR.T0SZ is set to the provided value. diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index a1c29d64e875..545b5d8976f4 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -80,26 +80,42 @@ * x19 primary_entry() .. start_kernel() whether we entered with the MMU on * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 - * x22 create_idmap() .. start_kernel() ID map VA of the DT blob * x25 primary_entry() .. start_kernel() supported VA size - * x28 create_idmap() callee preserved temp register */ SYM_CODE_START(primary_entry) bl record_mmu_state bl preserve_boot_args - bl create_idmap + + adrp x1, early_init_stack + mov sp, x1 + mov x29, xzr + adrp x0, init_idmap_pg_dir + bl __pi_create_init_idmap + + /* + * If the page tables have been populated with non-cacheable + * accesses (MMU disabled), invalidate those tables again to + * remove any speculatively loaded cache lines. + */ + cbnz x19, 0f + dmb sy + mov x1, x0 // end of used region + adrp x0, init_idmap_pg_dir + adr_l x2, dcache_inval_poc + blr x2 + b 1f /* * If we entered with the MMU and caches on, clean the ID mapped part * of the primary boot code to the PoC so we can safely execute it with * the MMU off. */ - cbz x19, 0f - adrp x0, __idmap_text_start +0: adrp x0, __idmap_text_start adr_l x1, __idmap_text_end adr_l x2, dcache_clean_poc blr x2 -0: mov x0, x19 + +1: mov x0, x19 bl init_kernel_el // w0=cpu_boot_mode mov x20, x0 @@ -175,238 +191,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) ret SYM_CODE_END(preserve_boot_args) -/* - * Macro to populate page table entries, these entries can be pointers to the next level - * or last level entries pointing to physical memory. - * - * tbl: page table address - * rtbl: pointer to page table or physical memory - * index: start index to write - * eindex: end index to write - [index, eindex] written to - * flags: flags for pagetable entry to or in - * inc: increment to rtbl between each entry - * tmp1: temporary variable - * - * Preserves: tbl, eindex, flags, inc - * Corrupts: index, tmp1 - * Returns: rtbl - */ - .macro populate_entries, tbl, rtbl, index, eindex, flags, inc, tmp1 -.Lpe\@: phys_to_pte \tmp1, \rtbl - orr \tmp1, \tmp1, \flags // tmp1 = table entry - str \tmp1, [\tbl, \index, lsl #3] - add \rtbl, \rtbl, \inc // rtbl = pa next level - add \index, \index, #1 - cmp \index, \eindex - b.ls .Lpe\@ - .endm - -/* - * Compute indices of table entries from virtual address range. If multiple entries - * were needed in the previous page table level then the next page table level is assumed - * to be composed of multiple pages. (This effectively scales the end index). - * - * vstart: virtual address of start of range - * vend: virtual address of end of range - we map [vstart, vend] - * shift: shift used to transform virtual address into index - * order: #imm 2log(number of entries in page table) - * istart: index in table corresponding to vstart - * iend: index in table corresponding to vend - * count: On entry: how many extra entries were required in previous level, scales - * our end index. - * On exit: returns how many extra entries required for next page table level - * - * Preserves: vstart, vend - * Returns: istart, iend, count - */ - .macro compute_indices, vstart, vend, shift, order, istart, iend, count - ubfx \istart, \vstart, \shift, \order - ubfx \iend, \vend, \shift, \order - add \iend, \iend, \count, lsl \order - sub \count, \iend, \istart - .endm - -/* - * Map memory for specified virtual address range. Each level of page table needed supports - * multiple entries. If a level requires n entries the next page table level is assumed to be - * formed from n pages. - * - * tbl: location of page table - * rtbl: address to be used for first level page table entry (typically tbl + PAGE_SIZE) - * vstart: virtual address of start of range - * vend: virtual address of end of range - we map [vstart, vend - 1] - * flags: flags to use to map last level entries - * phys: physical address corresponding to vstart - physical memory is contiguous - * order: #imm 2log(number of entries in PGD table) - * - * If extra_shift is set, an extra level will be populated if the end address does - * not fit in 'extra_shift' bits. This assumes vend is in the TTBR0 range. - * - * Temporaries: istart, iend, tmp, count, sv - these need to be different registers - * Preserves: vstart, flags - * Corrupts: tbl, rtbl, vend, istart, iend, tmp, count, sv - */ - .macro map_memory, tbl, rtbl, vstart, vend, flags, phys, order, istart, iend, tmp, count, sv, extra_shift - sub \vend, \vend, #1 - add \rtbl, \tbl, #PAGE_SIZE - mov \count, #0 - - .ifnb \extra_shift - tst \vend, #~((1 << (\extra_shift)) - 1) - b.eq .L_\@ - compute_indices \vstart, \vend, #\extra_shift, #(PAGE_SHIFT - 3), \istart, \iend, \count - mov \sv, \rtbl - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - .endif -.L_\@: - compute_indices \vstart, \vend, #PGDIR_SHIFT, #\order, \istart, \iend, \count - mov \sv, \rtbl - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - -#if SWAPPER_PGTABLE_LEVELS > 3 - compute_indices \vstart, \vend, #PUD_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count - mov \sv, \rtbl - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv -#endif - -#if SWAPPER_PGTABLE_LEVELS > 2 - compute_indices \vstart, \vend, #SWAPPER_TABLE_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count - mov \sv, \rtbl - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv -#endif - - compute_indices \vstart, \vend, #SWAPPER_BLOCK_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count - bic \rtbl, \phys, #SWAPPER_BLOCK_SIZE - 1 - populate_entries \tbl, \rtbl, \istart, \iend, \flags, #SWAPPER_BLOCK_SIZE, \tmp - .endm - -/* - * Remap a subregion created with the map_memory macro with modified attributes - * or output address. The entire remapped region must have been covered in the - * invocation of map_memory. - * - * x0: last level table address (returned in first argument to map_memory) - * x1: start VA of the existing mapping - * x2: start VA of the region to update - * x3: end VA of the region to update (exclusive) - * x4: start PA associated with the region to update - * x5: attributes to set on the updated region - * x6: order of the last level mappings - */ -SYM_FUNC_START_LOCAL(remap_region) - sub x3, x3, #1 // make end inclusive - - // Get the index offset for the start of the last level table - lsr x1, x1, x6 - bfi x1, xzr, #0, #PAGE_SHIFT - 3 - - // Derive the start and end indexes into the last level table - // associated with the provided region - lsr x2, x2, x6 - lsr x3, x3, x6 - sub x2, x2, x1 - sub x3, x3, x1 - - mov x1, #1 - lsl x6, x1, x6 // block size at this level - - populate_entries x0, x4, x2, x3, x5, x6, x7 - ret -SYM_FUNC_END(remap_region) - -SYM_FUNC_START_LOCAL(create_idmap) - mov x28, lr - /* - * The ID map carries a 1:1 mapping of the physical address range - * covered by the loaded image, which could be anywhere in DRAM. This - * means that the required size of the VA (== PA) space is decided at - * boot time, and could be more than the configured size of the VA - * space for ordinary kernel and user space mappings. - * - * There are three cases to consider here: - * - 39 <= VA_BITS < 48, and the ID map needs up to 48 VA bits to cover - * the placement of the image. In this case, we configure one extra - * level of translation on the fly for the ID map only. (This case - * also covers 42-bit VA/52-bit PA on 64k pages). - * - * - VA_BITS == 48, and the ID map needs more than 48 VA bits. This can - * only happen when using 64k pages, in which case we need to extend - * the root level table rather than add a level. Note that we can - * treat this case as 'always extended' as long as we take care not - * to program an unsupported T0SZ value into the TCR register. - * - * - Combinations that would require two additional levels of - * translation are not supported, e.g., VA_BITS==36 on 16k pages, or - * VA_BITS==39/4k pages with 5-level paging, where the input address - * requires more than 47 or 48 bits, respectively. - */ -#if (VA_BITS < 48) -#define IDMAP_PGD_ORDER (VA_BITS - PGDIR_SHIFT) -#define EXTRA_SHIFT (PGDIR_SHIFT + PAGE_SHIFT - 3) - - /* - * If VA_BITS < 48, we have to configure an additional table level. - * First, we have to verify our assumption that the current value of - * VA_BITS was chosen such that all translation levels are fully - * utilised, and that lowering T0SZ will always result in an additional - * translation level to be configured. - */ -#if VA_BITS != EXTRA_SHIFT -#error "Mismatch between VA_BITS and page size/number of translation levels" -#endif -#else -#define IDMAP_PGD_ORDER (PHYS_MASK_SHIFT - PGDIR_SHIFT) -#define EXTRA_SHIFT - /* - * If VA_BITS == 48, we don't have to configure an additional - * translation level, but the top-level table has more entries. - */ -#endif - adrp x0, init_idmap_pg_dir - adrp x3, _text - adrp x6, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE - mov_q x7, SWAPPER_RX_MMUFLAGS - - map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT - - /* Remap [.init].data, BSS and the kernel page tables r/w in the ID map */ - adrp x1, _text - adrp x2, __initdata_begin - adrp x3, _end - bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 - mov_q x5, SWAPPER_RW_MMUFLAGS - mov x6, #SWAPPER_BLOCK_SHIFT - bl remap_region - - /* Remap the FDT after the kernel image */ - adrp x1, _text - adrp x22, _end + SWAPPER_BLOCK_SIZE - bic x2, x22, #SWAPPER_BLOCK_SIZE - 1 - bfi x22, x21, #0, #SWAPPER_BLOCK_SHIFT // remapped FDT address - add x3, x2, #MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE - bic x4, x21, #SWAPPER_BLOCK_SIZE - 1 - mov_q x5, SWAPPER_RW_MMUFLAGS - mov x6, #SWAPPER_BLOCK_SHIFT - bl remap_region - - /* - * Since the page tables have been populated with non-cacheable - * accesses (MMU disabled), invalidate those tables again to - * remove any speculatively loaded cache lines. - */ - cbnz x19, 0f // skip cache invalidation if MMU is on - dmb sy - - adrp x0, init_idmap_pg_dir - adrp x1, init_idmap_pg_end - bl dcache_inval_poc -0: ret x28 -SYM_FUNC_END(create_idmap) - /* * Initialize CPU registers with task-specific and cpu-specific context. * @@ -729,11 +513,6 @@ SYM_FUNC_START_LOCAL(__no_granule_support) SYM_FUNC_END(__no_granule_support) SYM_FUNC_START_LOCAL(__primary_switch) - mrs x1, tcr_el1 - mov x2, #64 - VA_BITS - tcr_set_t0sz x1, x2 - msr tcr_el1, x1 - adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu @@ -742,7 +521,7 @@ SYM_FUNC_START_LOCAL(__primary_switch) mov sp, x1 mov x29, xzr mov x0, x20 // pass the full boot status - mov x1, x22 // pass the low FDT mapping + mov x1, x21 // pass the FDT bl __pi_early_map_kernel // Map and relocate the kernel ldr x8, =__primary_switched diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index e566b32f9c22..941a14c05184 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -52,6 +52,7 @@ PROVIDE(__pi_cavium_erratum_27456_cpus = cavium_erratum_27456_cpus); PROVIDE(__pi__ctype = _ctype); PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); +PROVIDE(__pi_init_idmap_pg_dir = init_idmap_pg_dir); PROVIDE(__pi_init_pg_dir = init_pg_dir); PROVIDE(__pi_init_pg_end = init_pg_end); diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 8c2f80a46b93..4393b41f0b71 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -11,6 +11,9 @@ KBUILD_CFLAGS := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) -fpie \ -fno-asynchronous-unwind-tables -fno-unwind-tables \ $(call cc-option,-fno-addrsig) +# this code may run with the MMU off so disable unaligned accesses +CFLAGS_map_range.o += -mstrict-align + # remove SCS flags from all objects in this directory KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS)) # disable LTO diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index afb6a337fa48..a9aad3a8b575 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -129,6 +129,22 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) } } +static void __init map_fdt(u64 fdt) +{ + static u8 ptes[INIT_IDMAP_FDT_SIZE] __initdata __aligned(PAGE_SIZE); + u64 efdt = fdt + MAX_FDT_SIZE; + u64 ptep = (u64)ptes; + + /* + * Map up to MAX_FDT_SIZE bytes, but avoid overlap with + * the kernel image. + */ + map_range(&ptep, fdt, (u64)_text > fdt ? min((u64)_text, efdt) : efdt, + fdt, PAGE_KERNEL, IDMAP_ROOT_LEVEL, + (pte_t *)init_idmap_pg_dir, false, 0); + dsb(ishst); +} + asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) { static char const chosen_str[] __initconst = "/chosen"; @@ -137,6 +153,8 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) int root_level = 4 - CONFIG_PGTABLE_LEVELS; int chosen; + map_fdt((u64)fdt); + /* Clear BSS and the initial page tables */ memset(__bss_start, 0, (u64)init_pg_end - (u64)__bss_start); diff --git a/arch/arm64/kernel/pi/map_range.c b/arch/arm64/kernel/pi/map_range.c index c31feda18f47..79e4f6a2efe1 100644 --- a/arch/arm64/kernel/pi/map_range.c +++ b/arch/arm64/kernel/pi/map_range.c @@ -86,3 +86,15 @@ void __init map_range(u64 *pte, u64 start, u64 end, u64 pa, pgprot_t prot, tbl++; } } + +asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir) +{ + u64 ptep = (u64)pg_dir + PAGE_SIZE; + + map_range(&ptep, (u64)_stext, (u64)__initdata_begin, (u64)_stext, + PAGE_KERNEL_ROX, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); + map_range(&ptep, (u64)__initdata_begin, (u64)_end, (u64)__initdata_begin, + PAGE_KERNEL, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); + + return ptep; +} diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h index 8d3d3ba44c27..04a1f576baee 100644 --- a/arch/arm64/kernel/pi/pi.h +++ b/arch/arm64/kernel/pi/pi.h @@ -17,6 +17,8 @@ static inline void *prel64_to_pointer(const prel64_t *offset) extern bool dynamic_scs_is_enabled; +extern pgd_t init_idmap_pg_dir[]; + void init_feature_override(u64 boot_status, const void *fdt, int chosen); u64 kaslr_early_init(void *fdt, int chosen); void relocate_kernel(u64 offset); @@ -26,3 +28,5 @@ void map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, int level, pte_t *tbl, bool may_use_cont, u64 va_offset); asmlinkage void early_map_kernel(u64 boot_status, void *fdt); + +asmlinkage u64 create_init_idmap(pgd_t *pgd); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 94847dbd31cd..cb6b756dbb56 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -45,8 +45,6 @@ #define NO_CONT_MAPPINGS BIT(1) #define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ -int idmap_t0sz __ro_after_init; - #if VA_BITS > 48 u64 vabits_actual __ro_after_init = VA_BITS_MIN; EXPORT_SYMBOL(vabits_actual); @@ -790,8 +788,6 @@ void __init paging_init(void) pgd_t *pgdp = pgd_set_fixmap(__pa_symbol(swapper_pg_dir)); extern pgd_t init_idmap_pg_dir[]; - idmap_t0sz = 63UL - __fls(__pa_symbol(_end) | GENMASK(VA_BITS_MIN - 1, 0)); - map_kernel(pgdp); map_mem(pgdp); @@ -806,7 +802,6 @@ void __init paging_init(void) memblock_allow_resize(); create_idmap(); - idmap_t0sz = TCR_T0SZ(IDMAP_VA_BITS); } #ifdef CONFIG_MEMORY_HOTPLUG diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 47ede52bb900..55c366dbda8f 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -200,7 +200,8 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 -#define KPTI_NG_PTE_FLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS | PTE_WRITE) +#define KPTI_NG_PTE_FLAGS (PTE_ATTRINDX(MT_NORMAL) | PTE_TYPE_PAGE | \ + PTE_AF | PTE_SHARED | PTE_UXN | PTE_WRITE) .pushsection ".idmap.text", "a" From patchwork Fri Nov 24 10:19:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467456 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 60EF1C61D97 for ; Fri, 24 Nov 2023 10:23:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=48Cnm8/pcFfA8uVfo2aTiU1WJdvPL2jxZUskUmS6Raw=; b=LD21ApuQ9HoWv/shs8xL/IbHg4 c4LeNDXp/q2rLlCFPlnhw+EFT79Rd1V0mutf3WKcIQcoNo/Or1rQUeuv7Wi8656SpbB+lqJIOzIzo zP4rDLgERILOJvbGflUs8SA/kji5nkcAuSkBQsczgmOHEgha3VRvEumnxd7QUSlqbnLaGO7j8hqUM 6nzQNVvC8ko7pgik8B3WlO/QIS2DkzBdVH/QJVSJmzcZ/IxhRu1yBOGEcAdL19tJnLtne+QIVhYdj kj8fddWBJwe9h59ROFXLXIFBgtngICLjMaAslGsbzW+wRH2J2MoP+aXsOgS+fF3r1ItkAv0uBr9ND 7On0zWNg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TL4-006pvn-0U; Fri, 24 Nov 2023 10:23:10 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TK5-006pZT-38 for linux-arm-kernel@bombadil.infradead.org; Fri, 24 Nov 2023 10:22:10 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=3dnhOg/Z1S19RkFfJIGILiqGdJNmDmPL6z9OV6/m1cY=; b=RvA4JW6s1E7f5yQns6ic7ldj4t Ak7Es3dtZRKWMl6m90cWXnqFAheDuDsbnPkgAOcHIz1QSUgQdZ+n9P79qipjKG20CvRTyBYdHj1bX syWrkd8lwcPvBUWmykF/wNzmXtNXpYkdi8jMNTUBLUhjGRuRtwO3XA196Zz+eIsQYgB+pN6FfxU09 1O4J5DHYJjj6ZA5iQPp+McSitGFxkglbFomi1lj7iS0EoSnk00pIPVCmUyw5rJBIEVJUkwqy6UjdY kluVi3NDWP7YlQDoCbZ1pvWI01rVQQ1dGMsdPg4iyBH/UmcI4zGsFIhd3sY/85OOgp0rSLzTfEcXY dUIJR+zw==; Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJu-00DqXo-19 for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:22:07 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-40b3712ef28so7536715e9.2 for ; Fri, 24 Nov 2023 02:21:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821298; x=1701426098; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3dnhOg/Z1S19RkFfJIGILiqGdJNmDmPL6z9OV6/m1cY=; b=D8gyXKmGt+mIuWf9M8qJzTp27CWIPHd6H9Z+whhTPzUHaduLosMa3e8sfL/QyrH085 THqrZO5iUW1b9N3Hmrk6q1VEV//L5XdGq77vbhf++EOHLFBTbDI92qgX9d+oq5TJrJzd 45zLQwPyRTLRzsVOeB+TZL8nLBvYQnxdKpi3rWZj3z8xIqf7eXM5Zb7UFGcZyLJVeaDG 3XssOm7loxGRRbnLgxcCH6oWz3DuJQhXYZnIFrQHPTRjOfyQAvQovFnzMOqA+bR27nl2 t24pgO+wijwKufgywkDzrXX6k/xKAh449Ra2KTyXJ3jNJsNvFAfgF5Jg7B61bscXFWc3 NQNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821298; x=1701426098; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3dnhOg/Z1S19RkFfJIGILiqGdJNmDmPL6z9OV6/m1cY=; b=hr/ccmNdwodz2GpauaeO1vFNTfIN9juFsjRzol8roxJTpxoC/gPfQ1pNioyjFWYzBI +GDTVpbdRvqRPidl3SfYIu5iBsNYpc9sLxLmQeTyVfvlTliNPSPnoU8Mxs9bD1ZItj1o MzD+Nwnevz9S5m8GeGFmTWVSjnY++ZrxTv2k2dBBTAymfhBytjexlJ71vvg9xuLvmFCS DuTvsbz1eCpZHzThy4yDoemo1Gjb8wzcG8Vfv+IssqsBEdI4+sXeBPGFNTYldDA+elGc npyHg1CECgcCMqqde3P7UbeCsQIN4qgdEDYm/A5dOTti264FiI5Mux9JSzIYGU1kBnXU d35A== X-Gm-Message-State: AOJu0YxSUIwR9rwZf/K9hivt9jrJ15yRRy9eSXUw2QsO1bwZ6G7qUI57 vfPf0CBogujUxJvThT4YB9oWV3PLOqDMMUAICodq/biOVQAYKOicjDLNmpAdgE+HcFoBqrCDAWD 67Hz1Su0nj5PVZ1qQccQM3oIwcv9aHJcK6m+z6hKJsNVNHWhkrMF9PdILttyrFV1vqllDWi8REx g= X-Google-Smtp-Source: AGHT+IGW+yf1LlDo138/89zmHrXlZ0yUx8r22/TDbeaS8rT6c5bR1yQjgsK1nX4Iv+e8nRiirAyFWYh3 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:1e23:b0:40b:2c9e:e144 with SMTP id ay35-20020a05600c1e2300b0040b2c9ee144mr45185wmb.1.1700821297871; Fri, 24 Nov 2023 02:21:37 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:17 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2126; i=ardb@kernel.org; h=from:subject; bh=fUAvuFuBmY0spPLjY96IzbCLIKESMxObqLpSTeEMtr4=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYtPvlesXrt5cNl/d+1XcDKmtDKpz/7Sa9CaF8bz8N O/yxZtZHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiy04w/JUVnn5xU8FrjztX 2Cb91VgUHKd2WNk/93f7pXRFg0rF+6EM/2s5rk3bPnf9mgtyua6q7+odLAtsxFbo7pNcVc6SIpO 4lgEA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-78-ardb@google.com> Subject: [PATCH v5 37/39] arm64: mm: avoid fixmap for early swapper_pg_dir updates From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_102158_646779_71230244 X-CRM114-Status: GOOD ( 13.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Early in the boot, when .rodata is still writable, we can poke swapper_pg_dir entries directly, and there is no need to go through the fixmap. After a future patch, we will enter the kernel with swapper_pg_dir already active, and early swapper_pg_dir updates for creating the fixmap page table hierarchy itself cannot go through the fixmap for obvious reaons. So let's keep track of whether rodata is writable, and update the descriptor directly in that case. As the same reasoning applies to early KASAN init, make the function noinstr as well. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/mmu.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index cb6b756dbb56..d36ba0811f5e 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -55,6 +55,8 @@ EXPORT_SYMBOL(kimage_voffset); u32 __boot_cpu_mode[] = { BOOT_CPU_MODE_EL2, BOOT_CPU_MODE_EL1 }; +static bool rodata_is_rw __ro_after_init = true; + /* * The booting CPU updates the failed status @__early_cpu_boot_status, * with MMU turned off. @@ -71,10 +73,21 @@ EXPORT_SYMBOL(empty_zero_page); static DEFINE_SPINLOCK(swapper_pgdir_lock); static DEFINE_MUTEX(fixmap_lock); -void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) +void noinstr set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) { pgd_t *fixmap_pgdp; + /* + * Don't bother with the fixmap if swapper_pg_dir is still mapped + * writable in the kernel mapping. + */ + if (rodata_is_rw) { + WRITE_ONCE(*pgdp, pgd); + dsb(ishst); + isb(); + return; + } + spin_lock(&swapper_pgdir_lock); fixmap_pgdp = pgd_set_fixmap(__pa_symbol(pgdp)); WRITE_ONCE(*fixmap_pgdp, pgd); @@ -628,6 +641,7 @@ void mark_rodata_ro(void) * to cover NOTES and EXCEPTION_TABLE. */ section_size = (unsigned long)__init_begin - (unsigned long)__start_rodata; + WRITE_ONCE(rodata_is_rw, false); update_mapping_prot(__pa_symbol(__start_rodata), (unsigned long)__start_rodata, section_size, PAGE_KERNEL_RO); From patchwork Fri Nov 24 10:19:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8DA63C61DF4 for ; Fri, 24 Nov 2023 10:23:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=1Goug3pQKwUUc9DN7+5ISjWsBsIe2zJNSWQMLZ9wujw=; b=mVhiKerFUbbaZXpTA/hTEsJfMD 0EAqDOu5wEkPB/29ppc4DHrpLobpweT/nY1lezWq/gqS1XovL1xH6oaPQ5Rin78QuGApHB6glLu31 PgUOLIil6r+MGNnrH9uIdbcqLbydOnqWAzPj7jGPhG8CZ6Qrtk2P3oRT9yKy/88sh5OrXT9cR85TH 4uZKopIL0CYw8Ia+NdbN0d0zcFELgFwLedukPMskockuaMFDfln+43MjSmP3CBwYRPLrWgvZ92V4d Dlcq4rshKMewZncWId1QhrUMOVd9SP9IZ4QdVMKz0rS1S/SDlvWqjd04y1X6ImaVWlUT9o4BPv2xf 9T2Ix2Xg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TKz-006psf-0N; Fri, 24 Nov 2023 10:23:05 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJi-006pNs-0o for linux-arm-kernel@bombadil.infradead.org; Fri, 24 Nov 2023 10:21:46 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=AVRRN4DuFbRb3wrfT0sNswvm342zPXmkt8+v7cdul60=; b=HZyc/PRqROWYgfP5My5HTW/VVn smvFCP1q+gAFiLhTF73yOJITwaMkjAGWOGEpnq4oXwdh/3PxmbIGtwdrq4tdq2/fhCIsedzrMKYy8 znWVT95hG0UzmqCrELCS3gzI0ygHYnAu89OsnAfM+TbCqeEGA6D4G6nNWYsUhDL25U0+KnDWJTFCp yp4vvRsEewNCMr1ftNp3oMVSV9n/se0KRRYh3IKHbfF0HWiSLpDXDgtubaPz9092zF9qtLC6GqEbr xO1qX9bsBb/5z+QGzkHNOUjQZU7iHi2NXyeFy0SSUNF2FJzqBHh4oVib2zFt8S1g57/YvM3E/SGRY XRlg/bXw==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1r6TJe-008SuG-Gp for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:45 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5ae5b12227fso22947017b3.0 for ; Fri, 24 Nov 2023 02:21:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821300; x=1701426100; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AVRRN4DuFbRb3wrfT0sNswvm342zPXmkt8+v7cdul60=; b=te8t9m6Gk+4ZKgGpV/TguoGSGwyjiSDv2nlZapKfWE8P9DIIr0+W8rNL+yOJYQ5j6l dZbrZqMre3+NX8V6te7LC5rA5o7czF20DP0MU7mbiV7e/dGp8Mk78gBoS+tY3Bm50o2e vYKj9iZcEE1Ha1pYygtJ1GxsotPd7qW9Wio/BWhuM8oxJi+IybYWv4Xp6Kq4y6R9BEuK JuYzhH/iXG+GMgj31IeHbZv87AXKg0Xoa+F1ZT8lNw7DKrasObBKVb0VL0Oe4dkghlEV NecVqOYh6jhwcIOuZSejz3un+rac/VMJOwuUVlQFF7s3Qjqnw3R1gIQJHlouqI2Vh3iT 7G6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821300; x=1701426100; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AVRRN4DuFbRb3wrfT0sNswvm342zPXmkt8+v7cdul60=; b=plrhBD3W0zPNF/qbhN1fT8hVpK83pq5ZxcHf900LPCS5Bbyqqj/jQ4p6RCa2iZaqLU eFFSHYqjlT8q4jHctNf7uZouqnz4l7DRKUaENDHSRhPtL2kGaiTD5PGyKRHo+Rov70aw OefpDwItVyF5QkpOjEucz5/Lsiqg+0jzyDr66NCvePAqEzh0gWHEqfzSkygQZQeQlwa5 CZT03GAbj+MPqNP3wQlcQblPeIN+T/d5y+xc2Xn5HLLUYVMTw9mqx37LSFdUPySRWS7R Jd9/UbPhw9E7EN7c2Kgmvp7O7LqYJwmqnswm81wVU1C2YSg1fazveUo4R0ry0mBLe16R 3RSA== X-Gm-Message-State: AOJu0YyEsX/P222WPhyrfY4o5lNOFiP5GrSBV/XIVO9+OoZoBhp2fCym be7DJTIrVP53LCbGWraZxhuNnpzYqB0WrHr/0z9syszhNZsQSw0jeBs9D53Tgq9NoKSyEut7UvQ mfLGb1upTn6I/QELhqs9ExlUP288uBxvLB4tK3C+oFN3nGVct7azcF4LHWoljqV81kA0M9b4fCt o= X-Google-Smtp-Source: AGHT+IF8qokzwi8ONAOq1Gj44a7z5Urdn56cOdHGdCb/yja+TbjFXM72WNAyb32v6KcAw6EwIW00SOXv X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:ab8e:0:b0:db4:5db7:9519 with SMTP id v14-20020a25ab8e000000b00db45db79519mr56770ybi.3.1700821300151; Fri, 24 Nov 2023 02:21:40 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:18 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10407; i=ardb@kernel.org; h=from:subject; bh=Yx1U2ztevh8h0bIECIvTnyCuqgsmKWI951lILqnMOJs=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYrP15WsZYTaX/8oxVWrJ21sv+qvrcOCMyA+r9pTXC aJPl3l2lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgInsOcTIcNAj/TT3v5oTNpF9 FczLzihsOOGkN+vv9ZbuE0cCGJ17HjP801XO/HbSokX0ZrToouPTfEtmuITMSk/XCRA8qHjLRGA zOwA= X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-79-ardb@google.com> Subject: [PATCH v5 38/39] arm64: mm: omit redundant remap of kernel image From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_102142_626466_7626D6C2 X-CRM114-Status: GOOD ( 20.70 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that the early kernel mapping is created with all the right attributes and segment boundaries, there is no longer a need to recreate it and switch to it. This also means we no longer have to copy the kasan shadow or some parts of the fixmap from one set of page tables to the other. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/fixmap.h | 1 - arch/arm64/include/asm/kasan.h | 2 - arch/arm64/include/asm/mmu.h | 2 +- arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/map_kernel.c | 6 +- arch/arm64/mm/fixmap.c | 34 -------- arch/arm64/mm/kasan_init.c | 15 ---- arch/arm64/mm/mmu.c | 85 ++++---------------- 8 files changed, 21 insertions(+), 125 deletions(-) diff --git a/arch/arm64/include/asm/fixmap.h b/arch/arm64/include/asm/fixmap.h index 58c294a96676..8aabd45e9a13 100644 --- a/arch/arm64/include/asm/fixmap.h +++ b/arch/arm64/include/asm/fixmap.h @@ -100,7 +100,6 @@ enum fixed_addresses { #define FIXMAP_PAGE_IO __pgprot(PROT_DEVICE_nGnRE) void __init early_fixmap_init(void); -void __init fixmap_copy(pgd_t *pgdir); #define __early_set_fixmap __set_fixmap diff --git a/arch/arm64/include/asm/kasan.h b/arch/arm64/include/asm/kasan.h index 12d5f47f7dbe..ab52688ac4bd 100644 --- a/arch/arm64/include/asm/kasan.h +++ b/arch/arm64/include/asm/kasan.h @@ -36,12 +36,10 @@ void kasan_init(void); #define _KASAN_SHADOW_START(va) (KASAN_SHADOW_END - (1UL << ((va) - KASAN_SHADOW_SCALE_SHIFT))) #define KASAN_SHADOW_START _KASAN_SHADOW_START(vabits_actual) -void kasan_copy_shadow(pgd_t *pgdir); asmlinkage void kasan_early_init(void); #else static inline void kasan_init(void) { } -static inline void kasan_copy_shadow(pgd_t *pgdir) { } #endif #endif diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index d0b8b4b413b6..65977c7783c5 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -110,7 +110,7 @@ static inline bool kaslr_requires_kpti(void) } #define INIT_MM_CONTEXT(name) \ - .pgd = init_pg_dir, + .pgd = swapper_pg_dir, #endif /* !__ASSEMBLY__ */ #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 941a14c05184..e140c5bda90b 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -55,6 +55,7 @@ PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); PROVIDE(__pi_init_idmap_pg_dir = init_idmap_pg_dir); PROVIDE(__pi_init_pg_dir = init_pg_dir); PROVIDE(__pi_init_pg_end = init_pg_end); +PROVIDE(__pi_swapper_pg_dir = swapper_pg_dir); PROVIDE(__pi__text = _text); PROVIDE(__pi__stext = _stext); diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index a9aad3a8b575..d5b345d68f07 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -125,8 +125,12 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) text_prot, true, root_level); map_segment(init_pg_dir, NULL, va_offset, __inittext_begin, __inittext_end, text_prot, false, root_level); - dsb(ishst); } + + /* Copy the root page table to its final location */ + memcpy((void *)swapper_pg_dir + va_offset, init_pg_dir, PGD_SIZE); + dsb(ishst); + idmap_cpu_replace_ttbr1(swapper_pg_dir); } static void __init map_fdt(u64 fdt) diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index c0a3301203bd..9436a12e1882 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -167,37 +167,3 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) return dt_virt; } - -/* - * Copy the fixmap region into a new pgdir. - */ -void __init fixmap_copy(pgd_t *pgdir) -{ - if (!READ_ONCE(pgd_val(*pgd_offset_pgd(pgdir, FIXADDR_TOT_START)))) { - /* - * The fixmap falls in a separate pgd to the kernel, and doesn't - * live in the carveout for the swapper_pg_dir. We can simply - * re-use the existing dir for the fixmap. - */ - set_pgd(pgd_offset_pgd(pgdir, FIXADDR_TOT_START), - READ_ONCE(*pgd_offset_k(FIXADDR_TOT_START))); - } else if (CONFIG_PGTABLE_LEVELS > 3) { - pgd_t *bm_pgdp; - p4d_t *bm_p4dp; - pud_t *bm_pudp; - /* - * The fixmap shares its top level pgd entry with the kernel - * mapping. This can really only occur when we are running - * with 16k/4 levels, so we can simply reuse the pud level - * entry instead. - */ - BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES)); - bm_pgdp = pgd_offset_pgd(pgdir, FIXADDR_TOT_START); - bm_p4dp = p4d_offset(bm_pgdp, FIXADDR_TOT_START); - bm_pudp = pud_set_fixmap_offset(bm_p4dp, FIXADDR_TOT_START); - pud_populate(&init_mm, bm_pudp, lm_alias(bm_pmd)); - pud_clear_fixmap(); - } else { - BUG(); - } -} diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 555285ebd5af..dd91f5942fd2 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -184,21 +184,6 @@ static void __init kasan_map_populate(unsigned long start, unsigned long end, kasan_pgd_populate(start & PAGE_MASK, PAGE_ALIGN(end), node, false); } -/* - * Copy the current shadow region into a new pgdir. - */ -void __init kasan_copy_shadow(pgd_t *pgdir) -{ - pgd_t *pgdp, *pgdp_new, *pgdp_end; - - pgdp = pgd_offset_k(KASAN_SHADOW_START); - pgdp_end = pgd_offset_k(KASAN_SHADOW_END); - pgdp_new = pgd_offset_pgd(pgdir, KASAN_SHADOW_START); - do { - set_pgd(pgdp_new, READ_ONCE(*pgdp)); - } while (pgdp++, pgdp_new++, pgdp != pgdp_end); -} - static void __init clear_pgds(unsigned long start, unsigned long end) { diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index d36ba0811f5e..bb9e13d54a5d 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -648,9 +648,9 @@ void mark_rodata_ro(void) debug_checkwx(); } -static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, - pgprot_t prot, struct vm_struct *vma, - int flags, unsigned long vm_flags) +static void __init declare_vma(struct vm_struct *vma, + void *va_start, void *va_end, + unsigned long vm_flags) { phys_addr_t pa_start = __pa_symbol(va_start); unsigned long size = va_end - va_start; @@ -658,9 +658,6 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, BUG_ON(!PAGE_ALIGNED(pa_start)); BUG_ON(!PAGE_ALIGNED(size)); - __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, prot, - early_pgtable_alloc, flags); - if (!(vm_flags & VM_NO_GUARD)) size += PAGE_SIZE; @@ -673,12 +670,12 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, vm_area_add_early(vma); } +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 static pgprot_t kernel_exec_prot(void) { return rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC; } -#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 static int __init map_entry_trampoline(void) { int i; @@ -710,60 +707,17 @@ core_initcall(map_entry_trampoline); #endif /* - * Open coded check for BTI, only for use to determine configuration - * for early mappings for before the cpufeature code has run. - */ -static bool arm64_early_this_cpu_has_bti(void) -{ - u64 pfr1; - - if (!IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) - return false; - - pfr1 = __read_sysreg_by_encoding(SYS_ID_AA64PFR1_EL1); - return cpuid_feature_extract_unsigned_field(pfr1, - ID_AA64PFR1_EL1_BT_SHIFT); -} - -/* - * Create fine-grained mappings for the kernel. + * Declare the VMA areas for the kernel */ -static void __init map_kernel(pgd_t *pgdp) +static void __init declare_kernel_vmas(void) { - static struct vm_struct vmlinux_text, vmlinux_rodata, vmlinux_inittext, - vmlinux_initdata, vmlinux_data; - - /* - * External debuggers may need to write directly to the text - * mapping to install SW breakpoints. Allow this (only) when - * explicitly requested with rodata=off. - */ - pgprot_t text_prot = kernel_exec_prot(); - - /* - * If we have a CPU that supports BTI and a kernel built for - * BTI then mark the kernel executable text as guarded pages - * now so we don't have to rewrite the page tables later. - */ - if (arm64_early_this_cpu_has_bti()) - text_prot = __pgprot_modify(text_prot, PTE_GP, PTE_GP); + static struct vm_struct vmlinux_seg[KERNEL_SEGMENT_COUNT]; - /* - * Only rodata will be remapped with different permissions later on, - * all other segments are allowed to use contiguous mappings. - */ - map_kernel_segment(pgdp, _stext, _etext, text_prot, &vmlinux_text, 0, - VM_NO_GUARD); - map_kernel_segment(pgdp, __start_rodata, __inittext_begin, PAGE_KERNEL, - &vmlinux_rodata, NO_CONT_MAPPINGS, VM_NO_GUARD); - map_kernel_segment(pgdp, __inittext_begin, __inittext_end, text_prot, - &vmlinux_inittext, 0, VM_NO_GUARD); - map_kernel_segment(pgdp, __initdata_begin, __initdata_end, PAGE_KERNEL, - &vmlinux_initdata, 0, VM_NO_GUARD); - map_kernel_segment(pgdp, _data, _end, PAGE_KERNEL, &vmlinux_data, 0, 0); - - fixmap_copy(pgdp); - kasan_copy_shadow(pgdp); + declare_vma(&vmlinux_seg[0], _stext, _etext, VM_NO_GUARD); + declare_vma(&vmlinux_seg[1], __start_rodata, __inittext_begin, VM_NO_GUARD); + declare_vma(&vmlinux_seg[2], __inittext_begin, __inittext_end, VM_NO_GUARD); + declare_vma(&vmlinux_seg[3], __initdata_begin, __initdata_end, VM_NO_GUARD); + declare_vma(&vmlinux_seg[4], _data, _end, 0); } void __pi_map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, @@ -799,23 +753,12 @@ static void __init create_idmap(void) void __init paging_init(void) { - pgd_t *pgdp = pgd_set_fixmap(__pa_symbol(swapper_pg_dir)); - extern pgd_t init_idmap_pg_dir[]; - - map_kernel(pgdp); - map_mem(pgdp); - - pgd_clear_fixmap(); - - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), init_idmap_pg_dir); - init_mm.pgd = swapper_pg_dir; - - memblock_phys_free(__pa_symbol(init_pg_dir), - __pa_symbol(init_pg_end) - __pa_symbol(init_pg_dir)); + map_mem(swapper_pg_dir); memblock_allow_resize(); create_idmap(); + declare_kernel_vmas(); } #ifdef CONFIG_MEMORY_HOTPLUG From patchwork Fri Nov 24 10:19:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467461 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3B632C61DF4 for ; Fri, 24 Nov 2023 10:23:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=MyhLB6x4VjGqvbyeWJjUj0EBvWrfFBxyvCQgVOiiemw=; b=2UgYQ8TU2O/u06jo2B9K7v9vvK XY1OoW5S/YDOgg9kmSU0naUa0LrKnoFvLf58GfXQ/0lxtWlB0FTOmZqNRPDuqmHchTvJYN/XVND62 AzeQPj/P1Meypa+QsXVztbxX4e9Nlf1a+t+isWbr4iX7MZkY3tZVd4rFup5kHwqdgStgNOCMUIKsc dI6SYm4E1bYrNE1nh0iRinWmjyzL7b5GhEmKIsYeoT4ehwH4PG3Y521M8ENK+vxEGEcKesB9MyfpZ c1+x34xY9d2Ljf7hE0WvbHJatQTfomcFOAzdjBFCWJGhPxOLrnhnbxl8067h/i71McmVVjp9muAUW WLiNs9oQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TL3-006pvM-1j; Fri, 24 Nov 2023 10:23:09 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJg-006pMB-1t for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:21:57 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-40b346a11d9so9864455e9.3 for ; Fri, 24 Nov 2023 02:21:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821302; x=1701426102; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=imcZ68l5aw0m3ocVj6+r1KqdZudjnVqQB3yjRHbNBSQ=; b=JnhW/nRuVV+NZHVf1t70isbxryKw0n5ez3e9eJD5SE9JK0CO8rR9mViVv3h0rpU4am Oi+8lai2D9UEIOrVOw+KIzs8puBUKgwliTS5qM9QmZICw+Qb15rQZR0fcuSrgS4n4XCI 8T1b3cTIRckPZwYtFx2Ylz0OI1dlDT+0TZ5Qp9tXxXPM84xjiRK+50Bw+ROkiD93qf+s bdmWF3jvIodf28I3TCFsCXjCBLzCMCxPfd23hxckE6Lz1JSenGpR3Xd6KDIknjkW68NA N/ikCHpxJrfN160//PRARFoBL59S9mSRl9vCatW9xH90rzP50o3l66hH+l51TgNNx71J EmCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821302; x=1701426102; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=imcZ68l5aw0m3ocVj6+r1KqdZudjnVqQB3yjRHbNBSQ=; b=s2FMzwa8QKO+2jjItBmYGoKhFn8Ecct8L+EuCbgbx55I2BTg8fUBxYCW2CoAjZI/KQ 0Rf/Tblmb8lpXENZLUqj0KIcIy2syzxCOE+6mDhUGneh8iUx1IGfK5i6gak8fsIOnJRd RzhV7zFDjHzbeSaoIbm+3JCA/GadTb+vNZGhTrHUS40rYpKv/Ea1iRMXK4RHgQplZqNk K3pGsloVfohaCnFXSdfwksyWNReL4qTUfhRnT8kKK1PDkwITYgLI5gRi0eyE4EvrsZPj jHTKjWJy9x16t6FDQSlr53X1PokobTnRnTEIQWAisqMPPgcq7fAGuXu0Q7Vtd+qzAF2D Txzg== X-Gm-Message-State: AOJu0YyJmP5zzVPZKTm6Gyh7/ajvqYU9v1shYMJf0iBX5KsvehbkWrfi 2Z4J+k21ovUMh2/xk2zvWdRcbB5F8PW7eR0a2BssB41MMcmMIBVKbwvvMzfDnPGk2S2o2gZHDdV 9PhT0lgitQTL8EAJg02DtDkek2sEVs0d5kWZb9+IwhpIxaSoA4uYHE3ffjrAdQBL5Hn/+r9hzKh g= X-Google-Smtp-Source: AGHT+IHUcIxlRO7ozBJcvDqw1GTNIJ7QqfNwE4kpvZJ5Ktcgo1rsNOvGidMQOzQ0HkcnQnP+YTepW7bh X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:4686:b0:402:f504:f2e9 with SMTP id p6-20020a05600c468600b00402f504f2e9mr44988wmo.0.1700821302424; Fri, 24 Nov 2023 02:21:42 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:19 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3478; i=ardb@kernel.org; h=from:subject; bh=qmsYzXP4m98BND8lXwIz1d4eg3pbd5m604LF8G46VZQ=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYovv5elREQZcK1YvNa7XWVL9ZZLSvoMd8Xl7Zyda3 XSueXWko5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEwk/BDDX5mbrh4aJWszjL4J 6fLp+q7m+/dIdkbLNROH/26CbW+rChkZTq7gyjhRntj26ESo6QJRhowrkvXv8i/Z2JhyTI1Z+qm KGQA= X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-80-ardb@google.com> Subject: [PATCH v5 39/39] arm64: Revert "mm: provide idmap pointer to cpu_replace_ttbr1()" From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_022144_653575_270FB9B4 X-CRM114-Status: GOOD ( 14.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel This reverts commit 1682c45b920643c, which is no longer needed now that we create the permanent kernel mapping directly during early boot. This is not a clean revert but the changes are straight-forward. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/mmu_context.h | 19 +++++++------------ arch/arm64/mm/kasan_init.c | 4 ++-- 2 files changed, 9 insertions(+), 14 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 3c487948b94e..d740e2ddd272 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -108,18 +108,13 @@ static inline void cpu_uninstall_idmap(void) cpu_switch_mm(mm->pgd, mm); } -static inline void __cpu_install_idmap(pgd_t *idmap) +static inline void cpu_install_idmap(void) { cpu_set_reserved_ttbr0(); local_flush_tlb_all(); cpu_set_idmap_tcr_t0sz(); - cpu_switch_mm(lm_alias(idmap), &init_mm); -} - -static inline void cpu_install_idmap(void) -{ - __cpu_install_idmap(idmap_pg_dir); + cpu_switch_mm(lm_alias(idmap_pg_dir), &init_mm); } /* @@ -150,7 +145,7 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, * avoiding the possibility of conflicting TLB entries being allocated. */ -static inline void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) +static inline void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp) { typedef void (ttbr_replace_func)(phys_addr_t); extern ttbr_replace_func idmap_cpu_replace_ttbr1; @@ -165,7 +160,7 @@ static inline void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); - __cpu_install_idmap(idmap); + cpu_install_idmap(); /* * We really don't want to take *any* exceptions while TTBR1 is @@ -180,17 +175,17 @@ static inline void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) static inline void cpu_enable_swapper_cnp(void) { - __cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir, true); + __cpu_replace_ttbr1(lm_alias(swapper_pg_dir), true); } -static inline void cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap) +static inline void cpu_replace_ttbr1(pgd_t *pgdp) { /* * Only for early TTBR1 replacement before cpucaps are finalized and * before we've decided whether to use CNP. */ WARN_ON(system_capabilities_finalized()); - __cpu_replace_ttbr1(pgdp, idmap, false); + __cpu_replace_ttbr1(pgdp, false); } /* diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index dd91f5942fd2..2ea0de4a8486 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -220,7 +220,7 @@ static void __init kasan_init_shadow(void) */ memcpy(tmp_pg_dir, swapper_pg_dir, sizeof(tmp_pg_dir)); dsb(ishst); - cpu_replace_ttbr1(lm_alias(tmp_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1(lm_alias(tmp_pg_dir)); clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); @@ -256,7 +256,7 @@ static void __init kasan_init_shadow(void) PAGE_KERNEL_RO)); memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE); - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1(lm_alias(swapper_pg_dir)); } static void __init kasan_init_depth(void)