From patchwork Tue Sep 12 14:15:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B0E09CA0EEE for ; Tue, 12 Sep 2023 14:17:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=qH2BN1ffkcGJVb0H17iJdfGITxRlomaa0cAR3BA/ur8=; b=wFvz0C0P4bFQbfLiGnMSEnzYNO HqhnIWXLtePotoRcWmLrYcRxe85LgJ79YN8UNRtZ60BlqeHlOD6eUV+2iIYXIy/QU+6IetWv+QeGg C2C4UNTi7ql67VfumSD+SjUCv/a8wqbiwez7VzDf/pdpllLe/EEbqKXNDf3+xFGTtC8yp5tLRFH6t pgxTgJrQP54Zl41whu9vldqON9BS1HKz3JDSLff4IWfM/7wp5PTwSuxZ3LByB/UcpGyPJnQpHHJh8 RZNkVgR7GKey58J5iHNt74/GSj2eI2hP2Utijta3MQKuEVeZBaRnIpaBI+VjEYVywsO7ZXU4sZ8Yg 4m7pYDUA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Cf-003Vp0-23; Tue, 12 Sep 2023 14:17:21 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Ca-003Vn9-1H for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:17:18 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-d7e81a07ea3so5496510276.2 for ; Tue, 12 Sep 2023 07:17:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528234; x=1695133034; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=snu7FzcVDFbhM35b3LBnQ/OHeKTO1p697+eImYuAVik=; b=4vQyDGtjRWSdGUf91vbESkRAxXKbVzfBKAv70H06zxaxdU43OjaYmTcZfWVH/eOAXH WMCVxnHERsxazAOc3OY4m4cWa1rVJwTFbYIXjQv2W4CSs07j+e+d+o6Yv1Lk5oNQWYw3 mDMs+Q1UO+LQ+87zlTtU45kuv+1ZwTzAAY/0yCG8qzrgDG8+g1IFWmJnXXARs5A1MEiq a6My1Ikojlp8lNyYGhL4sEXX0CQcwfVmgnlPiZhY39w7LqMwJplC4jG+9TzYqK9LYU4e fSpmcZoCc5lVza/0TDxryioMImMzuGk0Ltmtj3zxPfkOfFAF10LiMFaZimjiY6rLxdkc JQpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528234; x=1695133034; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=snu7FzcVDFbhM35b3LBnQ/OHeKTO1p697+eImYuAVik=; b=WHlVyTrzyMh1Pyc8SMEzVofQWrKa2z4hTkANiVD5g4O2zxUdtUQSJYQtEEbAbnUWoZ jScoXMWFKe6szP6dfc4rX3yBU6WaITm/raHJVUkROEyxfm4xkX6Ws5arPMTZ53MZy+nt 6b01nQ6P94LUMAFfHgQ3GecoXQ4NaP5BVgI7lRMVz1DAzZCzTLxoZap53jPfyuftQR+l Hum2OyXZeJDMwYYg7mUcwtQ0G3gmGZUY+NgL0PEDm9QzdDfnXb1s9DOTURD05JG0IEkB JjPaP3gM2gmcZjUo+1JQsENhrdbJlcNX+/kVFLzBbjVPateGIQ+4CopisdR3caeqsvxe 7cRA== X-Gm-Message-State: AOJu0YwwwR5ndk5jtt2TTcXTnr8nyeYqQHbZJA/Tz3IXtd83c+aSHMr4 6KLYpvVcEOy98LzWxSy5QNcxginf6oD69bS3G35Zl1Nk2KRUncg/6LDyrVfMiAzGAHBTpE/3T9Q 0wIf9rrtQ/KyDx5jYE8cn/YNRpPEVuxQj7XmDm+X4y4DcmPwN2EqstIlFrvzJruqi2kT7a1sFtc 0= X-Google-Smtp-Source: AGHT+IHuTMIzXqxIbNHetSku+tokVJYvWzcx7fxqSbJJHukoz7y6vlyuLcKd6kwyCPqvGvjXFnZDg75v X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:d295:0:b0:d7f:8e1f:6df2 with SMTP id j143-20020a25d295000000b00d7f8e1f6df2mr293240ybg.10.1694528234161; Tue, 12 Sep 2023 07:17:14 -0700 (PDT) Date: Tue, 12 Sep 2023 14:15:51 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1046; i=ardb@kernel.org; h=from:subject; bh=V746H3wtI2XKLyDLr9R3i/ElKXAcOwGhVEMFlEghmvc=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaPq6H3rbg1kv6d3MXKh2MmpeZWnV+tBTZzObIv/8E D/Y/r+po5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzk2H2G/153C9rK75VbrAh9 dPnM/P2TShY33AkIYVIp3eZ5g6PBbjvD/8ivT9TjffjnPl6y7JzljLUN/eGfC0NMZ9nyOmtd7eZ O4gQA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-64-ardb@google.com> Subject: [PATCH v4 01/61] arm64: kernel: Disable latent_entropy GCC plugin in early C runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071716_432598_F36AF991 X-CRM114-Status: GOOD ( 11.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In subsequent patches, mark portions of the early C code will be marked as __init. Unfortunarely, __init implies __latent_entropy, and this would result in the early C code being instrumented in an unsafe manner. Disable the latent entropy plugin for the early C code. Acked-by: Mark Rutland Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/pi/Makefile | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 4c0ea3cd4ea4..c844a0546d7f 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -3,6 +3,7 @@ KBUILD_CFLAGS := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) -fpie \ -Os -DDISABLE_BRANCH_PROFILING $(DISABLE_STACKLEAK_PLUGIN) \ + $(DISABLE_LATENT_ENTROPY_PLUGIN) \ $(call cc-option,-mbranch-protection=none) \ -I$(srctree)/scripts/dtc/libfdt -fno-stack-protector \ -include $(srctree)/include/linux/hidden.h \ From patchwork Tue Sep 12 14:15:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381756 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A6781CA0EEC for ; Tue, 12 Sep 2023 14:17:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=mKbAhV3Oir//GAiaBvdoV7Lg+caswN5t8qC4P0ii3OQ=; b=4YaEjpCZhDMDrdPQ/vI3xu1Fl2 WpW5z47dMlKSm2hRPag6vfUUqRxDfwZFzt8g8mToUntalXQrHU0nbPSqowDRVTMYADoXIEbwvtYZF jrjb+aHptIZm+VobczARKbO54gDlcqNjNIZXL9Y+H9n/m8bMpNbMM96FyiAI7yDf0hfO+9GKRLY7+ hgNh02MRjrwNleSx4UV7UltYuQZ6Lm65Se0acjAl0kBCKr5u7fzQZUc26ZGhUfa6n2Iy6iUxugkRh qPg0JjzHurzpWF6+pDpZ75dmszDS1YVu/+RfJWujZkLOQIso5S+aMe+C+NSsw6t6TZTIF9Ca/48Jb yFaq6DGA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Cg-003VpY-0Y; Tue, 12 Sep 2023 14:17:22 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Cc-003Vnx-2B for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:17:20 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-59b6083fa00so43625437b3.0 for ; Tue, 12 Sep 2023 07:17:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528236; x=1695133036; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rLlY/95tjuLc6IDcLl8sfILHPIdJZTxbO9x7CsV1Df0=; b=JELyO6xpjKRaNOUh+pQjSQ3dtoQnlXQxWNs7WyFD0Pc11F+6ETEjFruKzIjVReNdtl dcHrrC0pcjrwt7otu6H2NDYVcibZU4ZJx9PdyXqPlQaUm1rwhuav7h0/I5iaqmMy4QXe Iya+mkycIbd7uZLrZjOF+oBzEJieqkVZiYYDDpe6LlNZ3y/gAby9TQ/GskPYTv3a+bqD DVmBSAJsuQJpzqrHeRRQSCdT6bC3p2LG9c7HEhDLDv2zTzHJXTKy77PW3b8OgymgnrVI /dcJR9p20rNltvrr02okT5FGYufpDybImwbeUyR+NnR9ymBbDRRwkPJ0k5vx6VwrkWXZ Hpwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528236; x=1695133036; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rLlY/95tjuLc6IDcLl8sfILHPIdJZTxbO9x7CsV1Df0=; b=HtRTV36DGE/DqgT9trnBocrT5kPLgDmCUBPK8iyyCbz2n5sBFab5/90g5lvkAMDCwC c3NGMVn8Y6+22Z16NkcRdM5mQ6oWio6+gzqTMKtXdDmvtjbl4FQTlL9+v8+JXLCohNq4 3CSLX1T0aQiQjjQ2G3Q4oPDaUx28VhIb7nt1ttat18Lvre5XXyDsVveGJhRdYEG6PGE5 FJ8SnBf5jngn0PaaAOa6atKfS3mzECWIkkO7lDxRubGhnrFd4jOsoQ1/DVncOmWuUlYa AXWHjW2MmDChb7Uaf/yTnhAYl3gPnebauYYSVvwFM0RPbwVl+vrQW/uhy/a34g600St+ LqTw== X-Gm-Message-State: AOJu0YzUpUPZJPcy9rMbZX5XsbziqmI53EGrmTj6Bp0hJP05ktE+3RMo GLZRu/seuULB7XcVtsLAZWYrbSfTwKBYz9tfxoizC8FqT1RIYhC03NjoUbTp+OpbquzQG6rEOwj Rkl1MtMN2gdl9LdPXIbXWNKP2ODxJT7r/W0GOXOXSjQk7id7/B8n5SZ3Splk8XG9jeLsbGAdpHi w= X-Google-Smtp-Source: AGHT+IFVnpOTrt0nqnGJuDHHnxt+XyUEbf6Rcx27SMEyboc1HsYylrV+SZo3851BFUkajTuhKGZxEJkZ X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:ae48:0:b0:592:7a69:f61b with SMTP id g8-20020a81ae48000000b005927a69f61bmr320805ywk.0.1694528236561; Tue, 12 Sep 2023 07:17:16 -0700 (PDT) Date: Tue, 12 Sep 2023 14:15:52 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2827; i=ardb@kernel.org; h=from:subject; bh=Gmv54HGEg26w3b7wLxjCBwmFzhfBCXDT9Z81d9Krxfg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaEbLg3a7aRejeX/OSFX4tfRqwBGt5H6Jz2If783+5 id27DNjRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjI/dOMDLvPLer/z3jp147D aXurrs22vtvMKeB5emlZ6N/MyOiwhbcY/jsIBR+WLeFQ6Ew7NXP26errdvwXVMwXWae63305e8M PMX4A X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-65-ardb@google.com> Subject: [PATCH v4 02/61] arm64: mm: Take potential load offset into account when KASLR is off From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071718_711469_BEFDF27C X-CRM114-Status: GOOD ( 20.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We enable CONFIG_RELOCATABLE even when CONFIG_RANDOMIZE_BASE is disabled, and this permits the loader (i.e., EFI) to place the kernel anywhere in physical memory as long as the base address is 64k aligned. This means that the 'KASLR' case described in the header that defines the size of the statically allocated page tables could take effect even when CONFIG_RANDMIZE_BASE=n. So check for CONFIG_RELOCATABLE instead. Signed-off-by: Ard Biesheuvel Reviewed-by: Linus Walleij --- arch/arm64/include/asm/kernel-pgtable.h | 27 +++++--------------- 1 file changed, 6 insertions(+), 21 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 85d26143faa5..83ddb14b95a5 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -37,27 +37,12 @@ /* - * If KASLR is enabled, then an offset K is added to the kernel address - * space. The bottom 21 bits of this offset are zero to guarantee 2MB - * alignment for PA and VA. - * - * For each pagetable level of the swapper, we know that the shift will - * be larger than 21 (for the 4KB granule case we use section maps thus - * the smallest shift is actually 30) thus there is the possibility that - * KASLR can increase the number of pagetable entries by 1, so we make - * room for this extra entry. - * - * Note KASLR cannot increase the number of required entries for a level - * by more than one because it increments both the virtual start and end - * addresses equally (the extra entry comes from the case where the end - * address is just pushed over a boundary and the start address isn't). + * A relocatable kernel may execute from an address that differs from the one at + * which it was linked. In the worst case, its runtime placement may intersect + * with two adjacent PGDIR entries, which means that an additional page table + * may be needed at each subordinate level. */ - -#ifdef CONFIG_RANDOMIZE_BASE -#define EARLY_KASLR (1) -#else -#define EARLY_KASLR (0) -#endif +#define EXTRA_PAGE __is_defined(CONFIG_RELOCATABLE) #define SPAN_NR_ENTRIES(vstart, vend, shift) \ ((((vend) - 1) >> (shift)) - ((vstart) >> (shift)) + 1) @@ -83,7 +68,7 @@ + EARLY_PGDS((vstart), (vend), add) /* each PGDIR needs a next level page table */ \ + EARLY_PUDS((vstart), (vend), add) /* each PUD needs a next level page table */ \ + EARLY_PMDS((vstart), (vend), add)) /* each PMD needs a next level page table */ -#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end, EARLY_KASLR)) +#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE)) /* the initial ID map may need two extra pages if it needs to be extended */ #if VA_BITS < 48 From patchwork Tue Sep 12 14:15:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381762 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 874E1CA0EEB for ; Tue, 12 Sep 2023 14:17:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=f+UWmta6XJTknlIiOB73F8xALFtUqNz9nnrqqqltlio=; b=ACtwreIuJ2ZwQ5mavmoXY6qGEO aGVz1P7vHYCP7VRK697zIhpQ8uMlENXJ6RF5SOk+c1pLklUgMvx3ANvrHR1mG9bbI7RAZg3oIEQ4y hHPim65KzmY8/xyUfMfRR12BHdNLsLuDzvLvhOnpEvJkvVP8/2kSBUbNy4fY4MWny+Z1Bv9K5d1iF Gtr1plcxCilEtYmnzJSjnb4ff/9wUC44aOYNht/kqZjy97QbOhCoIUm1MohU5kxB4dTRocYIdmU6o u2x2xN5ep0NUwwtImExj1hl+V+8bqYisqq2p9J1J2Lz1Unt3EoF2M0r5hThVsztvJmYbn3Ps0j2yA ZghDlc/g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Ck-003VrI-1g; Tue, 12 Sep 2023 14:17:26 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Cf-003Voh-1v for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:17:22 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-31f9186eb8dso2018717f8f.1 for ; Tue, 12 Sep 2023 07:17:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528239; x=1695133039; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=eOb/FBDmij/2vNJpti6zX3gVXW4T8JnmlfG1c99rhJU=; b=vZiaj0PDn92YNgbXZKanQM7kW/EPsa3K+qN545vIQC9c17d1LQEqf7iuCwFqBVsg9t FXVgEbO4VUNU7do47jpoKaQdMFDOn3YBOCi1gDWyWGXfONZTB9dWmnsT22+SYR+lNz/F LgNS7Qdbfwxz8VdE2LjmEqbh+iKKWllpHDJ9dL6rmNYO/nI4gJNolcntC4mUpe0K2RJW iGZDjEKgMaWD9DfkmBbq5jeToyxAtfzR77UeLO36ImU0QkZNaKaVN4eTAnHkhzAW23U2 CzKHqfDlqbv+h54E7mw/pYgMXFEYO7TLCTFyEF8N3po/N37uX6w5yd1+HYoy9sTwyDP3 AXEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528239; x=1695133039; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=eOb/FBDmij/2vNJpti6zX3gVXW4T8JnmlfG1c99rhJU=; b=lrYB7je/fB9lecgmskM/YyVv0KC78gW/6c6fVXiH8ws+ujNtmVF5x4JZZ2iHJygY7t 881C+0ZspmV0rKvknewj9cmHyO/2n2frQaYcDLGGp8r68VNt0eEGDlTdKWNeQAeppVdu 0RJpsxP4vK50J028vjeVskEWUWqC1/bAqzEO6sbJ+KzIP8g+I+ESbPa4QSJfSnTaU0xp 4ictZ6+b6oQrpGxJtQ5TiMytc6f3Lp7rnbBtmsEZSCim8rb7gctF0vqy8ymPj0odHcn0 ZDvnXY3ku6wHpOmk3akhOeRyAACkf7kKb7B/cL+v0V7v4fAGP/0JMhczVJ2w4o2ZpFHa kznw== X-Gm-Message-State: AOJu0YwHo2MLlqSzHPvXN1SPu7wO6g7pJUga5rqG5m9NbdrO3RVBSNrZ qPFWpAv4Cd3Ww0aLoSxYo3ukuB0ltKSbaGQ6AAFZ1xAQUAj4giriwt8zVWFbbaa5DVt4u4VfmFL VAFij6iBlbMLRrDiT4dJ68q0fIkdgRufe7RLy4GAskvGu0mZwf8xxsSjlu4SpL1uW7uW2AHHqtZ Q= X-Google-Smtp-Source: AGHT+IH5Gjz1j9kZra4s09DIYfL/rr6h7oBCEP+Eg4GGTTg5wd80PLard/dGKqj6aPLuWyn6BXSKwlaV X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:f64c:0:b0:31a:ed75:75d5 with SMTP id x12-20020adff64c000000b0031aed7575d5mr147849wrp.6.1694528238991; Tue, 12 Sep 2023 07:17:18 -0700 (PDT) Date: Tue, 12 Sep 2023 14:15:53 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2451; i=ardb@kernel.org; h=from:subject; bh=ROSXFKE0XZd48MC38Ki7uymklvK1Vn6RW508I4T87NY=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaOZEpX1qDuVTbbQ6Tp1cu71i/sXVa11DRINK3rT3T jronM/bUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACay4hrDX8Ga9z7bCmaumFt4 8kV1oV3+47VFThzNcbKuTQ+WtdWqMDEydE3TWZjH1vz3sLpU8BerlUy/GEte3U9Qb/723mTm+4s 97AA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-66-ardb@google.com> Subject: [PATCH v4 03/61] arm64: mm: get rid of kimage_vaddr global variable From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071721_636895_C8DE6204 X-CRM114-Status: GOOD ( 16.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We store the address of _text in kimage_vaddr, but since commit 09e3c22a86f6889d ("arm64: Use a variable to store non-global mappings decision"), we no longer reference this variable from modules so we no longer need to export it. In fact, we don't need it at all so let's just get rid of it. Acked-by: Mark Rutland Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/memory.h | 6 ++---- arch/arm64/kernel/head.S | 2 +- arch/arm64/mm/mmu.c | 3 --- 3 files changed, 3 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index fde4186cc387..b8d726f951ae 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -182,6 +182,7 @@ #include #include #include +#include #if VA_BITS > 48 extern u64 vabits_actual; @@ -193,15 +194,12 @@ extern s64 memstart_addr; /* PHYS_OFFSET - the physical address of the start of memory. */ #define PHYS_OFFSET ({ VM_BUG_ON(memstart_addr & 1); memstart_addr; }) -/* the virtual base of the kernel image */ -extern u64 kimage_vaddr; - /* the offset between the kernel virtual and physical mappings */ extern u64 kimage_voffset; static inline unsigned long kaslr_offset(void) { - return kimage_vaddr - KIMAGE_VADDR; + return (u64)&_text - KIMAGE_VADDR; } #ifdef CONFIG_RANDOMIZE_BASE diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 7b236994f0e1..cab7f91949d8 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -482,7 +482,7 @@ SYM_FUNC_START_LOCAL(__primary_switched) str_l x21, __fdt_pointer, x5 // Save FDT pointer - ldr_l x4, kimage_vaddr // Save the offset between + adrp x4, _text // Save the offset between sub x4, x4, x0 // the kernel virtual and str_l x4, kimage_voffset, x5 // physical mappings diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 47781bec6171..49a49b37580b 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -52,9 +52,6 @@ u64 vabits_actual __ro_after_init = VA_BITS_MIN; EXPORT_SYMBOL(vabits_actual); #endif -u64 kimage_vaddr __ro_after_init = (u64)&_text; -EXPORT_SYMBOL(kimage_vaddr); - u64 kimage_voffset __ro_after_init; EXPORT_SYMBOL(kimage_voffset); From patchwork Tue Sep 12 14:15:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 38E2BCA0EED for ; Tue, 12 Sep 2023 14:17:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=vXC3QlPAVtLXrs7VZjasden5z3KQLg0ZzjgpQoH7Bnk=; b=iBhWv4HW7B3rAwwwDo/2zxjFDL q+p3fp4ixDPs/PyTisteEk8OTrxI1EfBmtKc+6LOv/MLHYPOlmVjuV2HJ8BbUyuzGdCAto4ph2p4K TOYf2P6edZjgyjLBIdSaaQIDA5fpYa/kUanU0ynkwDgBlvN/cR6aUH/wqEcxbPEFfL9kAtqytJBhN OakU8mc/u88qu0hMT2gkdsPWPwbFs4PZ2R/iaswFIxzhQlN3yIBpyTiq0k1HZFJSgumDo7JIFkvCp /p55eMx0GosOVMfv7ozFVS70n0431Vpy9mCKMj/k4GeH7i77euKuUAMRYRmoNlYlLHAkyjZHyVM/o mCxPjNRg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Cl-003Vrc-0D; Tue, 12 Sep 2023 14:17:27 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Ch-003VpT-2H for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:17:24 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-58d799aa369so67562007b3.0 for ; Tue, 12 Sep 2023 07:17:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528241; x=1695133041; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Kyf/ErPiip1ApkBGDB/QoNksVTDqpAoO4jDwl90YVbY=; b=Kkzdk79a1G/uF3f9MrBUgyXnsVlxdwu3Im6EQFrHQ77MpDviesq8n92q+ta5viz+n+ /KbNyBNvqxa40XhLE0r/Slf9EhCSUSIp9mpj076H1rIonlrcDIPX1XjMNgPopG8FTDfG xWu+3cYZ/2Y9u1RKLTbE4GgGNY+hlTCY+8DiknlOPQf5uLFHXTn+CLC1WvZyufxZQxrG ZnWzNkOAbKcd1WXCb8Ph+6xjKCaZdYJ6VaQqGR4VlrEUUWF0M3DxWosvaNdqrfsg09FP DAz2w1kOWUrCU+VzLC43R4IAOtZWscY3KtZc5WpP1FU42BCnNyrYd4qdX1qYpETCq39E Uhug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528241; x=1695133041; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Kyf/ErPiip1ApkBGDB/QoNksVTDqpAoO4jDwl90YVbY=; b=fXmTtmkdvqLikeA/pIHk9mqJ2lNqiOjlsb36OBGG40hMIIFjvHdypL/suX3W5lpgRR JW/8n46061fuSDVB9pYbisnhhFChB91XZPYC7J0Dq2Iri5i/KLCSFe8BjGfCMKg/+vJ4 +SGR1sipbBjaONMCZ65R8tjkLUj9HEeknOJSmV9fu/9m8Q10FLU1e4xyyubITZyGEYI/ byMUO6d0z7t1M/AsEnT37AF5ADqSQnIYR8eyYD50YTCH6lvHbZc2ANBD7PSBaBUN9bcR D74c8JNcYpgZvmZE1zVATN/r6RmPcijCRnkT4ipWAT/2mcxGJ6Zk8JDePWMnGP3mUw6W Ip6Q== X-Gm-Message-State: AOJu0YwEw8x5nXpa+5qHa6La58bLy0IJGPExe3Rd/3WGxRVUAVmPaUUr yFdIfAZnm3C0+A2D2kq6drEsBBXTo0rQPQGIGCOBEw9jaC07bFvil9tBCo0izIYToH1Oh9YFCa8 nR1zpDQp/7bBmz9WCc84LpapMv6A8HegftlfrIXdc1Y1YNsJKGZjbDfA0YeJMyPiv/oUEK+fbmT E= X-Google-Smtp-Source: AGHT+IGsMQs5BRzvvckE6UTsrKhuxuNS0gYdWHDRTz7KPhja+s8DSmjgzuaQZvoysANQYtO+gobJmzak X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:d002:0:b0:d7f:8e0a:4b3f with SMTP id h2-20020a25d002000000b00d7f8e0a4b3fmr311685ybg.3.1694528241619; Tue, 12 Sep 2023 07:17:21 -0700 (PDT) Date: Tue, 12 Sep 2023 14:15:54 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1731; i=ardb@kernel.org; h=from:subject; bh=LZS9DkdmR4a/TLcyjNXw6Rkmd/jkYDwqoR8m8lI9T4U=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaNYdvffiFYWXO+7c0DpdbijxklX5jvT/upl7th0tO LrF71BxRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjIzCKGv8J/xffsvnPn7vx8 3VVM2433zkgKrLvYmu414Yrz08ntj5cxMrzKvi2nFC+/W6zo8pbyiw2Ht0esqtzp6rbGqfJcn2W DFgcA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-67-ardb@google.com> Subject: [PATCH v4 04/61] arm64: mm: Move PCI I/O emulation region above the vmemmap region From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071723_741914_B5C1B066 X-CRM114-Status: GOOD ( 13.58 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Move the PCI I/O region above the vmemmap region in the kernel's VA space. This will permit us to reclaim the lower part of the vmemmap region for vmalloc/vmap allocations when running a 52-bit VA capable build on a 48-bit VA capable system. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/memory.h | 4 ++-- arch/arm64/mm/ptdump.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index b8d726f951ae..99caeff78e1a 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -49,8 +49,8 @@ #define MODULES_VSIZE (SZ_2G) #define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT))) #define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) -#define PCI_IO_END (VMEMMAP_START - SZ_8M) -#define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE) +#define PCI_IO_START (VMEMMAP_END + SZ_8M) +#define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE) #define FIXADDR_TOP (VMEMMAP_START - SZ_32M) #if VA_BITS > 48 diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index e305b6593c4e..d1df56d44f8a 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -47,10 +47,10 @@ static struct addr_marker address_markers[] = { { VMALLOC_END, "vmalloc() end" }, { FIXADDR_TOT_START, "Fixmap start" }, { FIXADDR_TOP, "Fixmap end" }, - { PCI_IO_START, "PCI I/O start" }, - { PCI_IO_END, "PCI I/O end" }, { VMEMMAP_START, "vmemmap start" }, { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, + { PCI_IO_START, "PCI I/O start" }, + { PCI_IO_END, "PCI I/O end" }, { -1, NULL }, }; From patchwork Tue Sep 12 14:15:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D77C4CA0EEF for ; Tue, 12 Sep 2023 14:17:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=dVsJOwWx6fgiuj91a/uCO/GrcVvfuVA712nCoBUxd/Y=; b=eJ32Zvq2jLxAJSpOUtWPS/83GX IBmfxSOxNAZV+TpmPe4wTWvR5IMQtMDldTmyAoHMCuvB2h0/MjstUM+0VKX3BTKOdH966Sb3TDjYG KkfsRa6/MBWVr384iGNQFXNJBEqdvfy3O7BQV4bd44WxgnkUPAwRhbZGLzhoEYA/HpjVH2lElPnE1 1BKM2/umq8qGFJbTYQhSIKglTFXCj/RU5qdKtUq/h5+Ts6xW0YFZbNG5BTqdPGSWtVnPLelgRCBO/ i4Cl/YGVn5288flJ9qjpzIQ22RaLCSbBz9SU6bWS0LmvMJTYMLRwpcLWB7JQ9OlC13SfKaSuVFiF2 fCPWlaTg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Cr-003VuT-0d; Tue, 12 Sep 2023 14:17:33 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Ck-003Vqi-0V for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:17:27 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-31c470305cfso3774544f8f.3 for ; Tue, 12 Sep 2023 07:17:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528244; x=1695133044; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FGjWmrp/LCWU7fgbtimbKfWIZkwRzJf0Hdf3dUgKZuI=; b=puNgTn/NAbmgBlMMQYr12SDkUFWRSilKCfR3q9I3caekqBN+Oh6x8LCzhapl6QO5PE /mG0zyr9Rrr2Q+GSs7Jr7Dkyk92OmAnv+85E4jFuSdb72R7xLLtpxTnxteQGsxgrJ5yF TD8MX4iVqM10sD/gf4Ovpclrk55eJKNvk5+5HMrqGiZAEFvBNLKx67CB8N++AxYDrGfu 8zpxwXsQUBdm+0ZG9gV4znUS+hPn1oAvH3ss02zgsRJtLYftd4BUQRwqUsC88abeJIqW B9NAE3eyKsSaWU+9+h7f7ULyLnSLpTyJMr+2rIBJ39YbPWQfJQMFjFrqTLrEkJGkWtPT DwAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528244; x=1695133044; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FGjWmrp/LCWU7fgbtimbKfWIZkwRzJf0Hdf3dUgKZuI=; b=ZHDuj6Hg4Li8Ky4tp5NYb6IyhY7PvvVEDgi5uDRmH41QVg0onP3D4DczHf97BVWqZJ /rR1R8ja8VJo15sBM6mA1WgloumLUrR6ArIIiHCitORBWxL78NSrkIk+5aGovnGuBbNi yoXmgHQxN9lmCmHSzoqCg1jU9us39adHM/9zkerQiOM3uGN2CKy8Yk9lq9F9tIJmN85E 1Dr6WfyvlxS9iZuzeWJk+pIabQVUqNCHFzhRzb793YXAhTPhKTeMMg4OayuKB4RiY4cS tix0I8NXNFpkcyFetN9ZOv+xdD4dD8suspPdxkRgUHYZplzu/sXgJ9JVNRd9FWJR0gYU +UBQ== X-Gm-Message-State: AOJu0YxEk439lBYUIyRdDCdlbKxDQ5309E6r6u5zQ8xTOVptYx0WbCZE kMI4fqGW7vJmVcsSTDlVbFXGD/mKdj6xZx1pdYwBs3GJCfcPwmM+CSeAoAz0kVGG02lqOMAcSC8 aqXXiFacVnD+W3slEyiWmHRD59VQNzonzxMSN/0mmGi4GWKb60j+dAAlIggI41pqHvOucEiUuIO s= X-Google-Smtp-Source: AGHT+IGFFC5O8+gMrQywD0Q5wAh+VnPsRyGgbLbWJ+mj8xp1nxE3EuVVUMf3XsWK9ESTMZprPMSAENTF X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:362:b0:31f:3049:5759 with SMTP id f2-20020a056000036200b0031f30495759mr154885wrf.1.1694528244400; Tue, 12 Sep 2023 07:17:24 -0700 (PDT) Date: Tue, 12 Sep 2023 14:15:55 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2524; i=ardb@kernel.org; h=from:subject; bh=TcKfFJ74di7EwtwuCY+EKlB4E2qjOZmy9HG1PFKjSg8=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaPbFA1tXqbRE7n6Z03UgeYPBqqp3c94cCFw268vJN 1VbXR7e7yhlYRDjYJAVU2QRmP333c7TE6VqnWfJwsxhZQIZwsDFKQATqd/L8IdX2LF8UZZov0XM yqucu4+s/ST8me/okyfLlXdfSStnsuhlZLihWyXqdGTpAwV7v1fBqfpC0yOd2vsvux660OGpPoW TgRUA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-68-ardb@google.com> Subject: [PATCH v4 05/61] arm64: mm: Move fixmap region above vmemmap region From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071726_196887_6597339B X-CRM114-Status: GOOD ( 16.14 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Move the fixmap region above the vmemmap region, so that the start of the vmemmap delineates the end of the region available for vmalloc and vmap allocations and the randomized placement of the kernel and modules. In a subsequent patch, we will take advantage of this to reclaim most of the vmemmap area when running a 52-bit VA capable build with 52-bit virtual addressing disabled at runtime. Note that the existing guard region of 256 MiB covers the fixmap and PCI I/O regions as well, so we can reduce it 8 MiB, which is what we use in other places too. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/memory.h | 2 +- arch/arm64/include/asm/pgtable.h | 2 +- arch/arm64/mm/ptdump.c | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 99caeff78e1a..2745bed8ae5b 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -51,7 +51,7 @@ #define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) #define PCI_IO_START (VMEMMAP_END + SZ_8M) #define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE) -#define FIXADDR_TOP (VMEMMAP_START - SZ_32M) +#define FIXADDR_TOP (-UL(SZ_8M)) #if VA_BITS > 48 #define VA_BITS_MIN (48) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7f7d9b1df4e5..01fddcc2ae4d 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -22,7 +22,7 @@ * and fixed mappings */ #define VMALLOC_START (MODULES_END) -#define VMALLOC_END (VMEMMAP_START - SZ_256M) +#define VMALLOC_END (VMEMMAP_START - SZ_8M) #define vmemmap ((struct page *)VMEMMAP_START - (memstart_addr >> PAGE_SHIFT)) diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index d1df56d44f8a..3958b008f908 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -45,12 +45,12 @@ static struct addr_marker address_markers[] = { { MODULES_END, "Modules end" }, { VMALLOC_START, "vmalloc() area" }, { VMALLOC_END, "vmalloc() end" }, - { FIXADDR_TOT_START, "Fixmap start" }, - { FIXADDR_TOP, "Fixmap end" }, { VMEMMAP_START, "vmemmap start" }, { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, { PCI_IO_START, "PCI I/O start" }, { PCI_IO_END, "PCI I/O end" }, + { FIXADDR_TOT_START, "Fixmap start" }, + { FIXADDR_TOP, "Fixmap end" }, { -1, NULL }, }; From patchwork Tue Sep 12 14:15:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381760 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2AB83CA0EEC for ; Tue, 12 Sep 2023 14:17:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=+Of0kPvyXFzFJsCCQAlmnFnE3e4IVfpbk6L9x6A4lGU=; b=TealpfsYhchwZqipyGHYpReY1Y VdSbRRtl/WTf7ZrMpax/0e89o55sA9rMuLMWt8Q4GjBCnn7noUXrVpH0plJJNnq9ZwA3zYR24LuDu hTlm0AtiM3evUqI3OXceHu9OQF7sBZQ4sAjUTMRo/Rm5DkJtt5fawBrQxyB/SVZREBVx9gmpsd3t+ nUmQg5QhhY2vb6Cem1sxkMU2vvfLRm2D2+Bz9uLL0Ax+CkQS5XJP0MlGfUYXdQwDdxNPYv8iwQbsJ KiA3l1Ou42YVkGn/VPHEo2UdDMqoD+TjuJ2nVjYW1Bal8FTup4AQDONy8WyjuwCc58IVFx2i4ur2Y bGXK88UQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Cr-003Vup-2M; Tue, 12 Sep 2023 14:17:33 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Cn-003Vrg-0T for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:17:30 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-31adc3ca07aso3769274f8f.2 for ; Tue, 12 Sep 2023 07:17:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528246; x=1695133046; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QOaB6U1iEfQSZsU7eRF+99qkcmj7tpJKL0EHxJklm5g=; b=epsLEmCpwt7oVZSexLbVKR5OmV7OgS6kVUt+duCWBG/3xOju3E5+8/w/xZej4hRbQd HLyqLplIvhluSsR1Uyf6yHx8VAvMehg66DAjVUfkxSDbsbF3Q2kOyTH+LOrylNKxHEce 7vvsOvZnYlXf1S+s/60Pp/0LwkKIQomOH1gWmkrXby1KQ8T097PHlbRXDAeCykrdUpAh 8F0J5Ynj/fHZ1Dt+zGmMn2BLCbty6iicGJocu3USn+3QEUP3oJY1cBkp8ddOZ0bZ82rK dDUwzThZFC/uFO/lbBU8uHZAUDGLaaCLMNNxGD5IrJb0UXg5OhAGVUtSLo4e9NTmU8xw Bwow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528246; x=1695133046; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QOaB6U1iEfQSZsU7eRF+99qkcmj7tpJKL0EHxJklm5g=; b=tESrDrUifZz6BqqSVdjvuyWfIVfALPiTei2+5vkF6LOVuAb7KiwzYAkep2FDT5vOp2 YB4VBxn4Bx+yJl+rs0J9v0/4SdnZLVtNT7UzkN9GADSkuK06z4J93UAZrJHpvmZ00ylS VE/6bQMTwNL1G+2WD6hcuWxO0Uvl48+UhWm6fNsLN3FJf346Uj1ilV/uT5NaERXqBwfE JwLkNCj3mNFQrI+iP7nTbPB2R3ETulChAW4194fZIEs8KhUlosDTGb10nIoFg6+t2Lt0 iIMP89npfRJOoozDaYgqrywniLdh5ExeG5pq4WBxKJdJ2Ht2pZqVeiEv9dmCn1MCaefC eSRA== X-Gm-Message-State: AOJu0YwGoIifZhJJwkcRSHweAA7JHWxndtMXNXaPxXsIoBjTit2XuBIA psJ+m8BBjLOOOWTbQqTs1yI21ugfqpdYZqqLw63Z7FEiizAqn81yoc4nmlkPmq9mnoIkYQLVyma 4bazaQCrHM+owwQekCfWrLX900AuFanZP1SdxIScPYnLdRYTtnwu3+9/oT9Cdl6zhc5clkhrCGr g= X-Google-Smtp-Source: AGHT+IG1vHZ9s7abuaNp6jiDk0T2Hsc81c3e63uJx3kcSAx5p5sUXYsLTPscpcvzFIXHt46YqYKCwe5m X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:5a0a:0:b0:31f:888b:9a4 with SMTP id bq10-20020a5d5a0a000000b0031f888b09a4mr127203wrb.12.1694528246584; Tue, 12 Sep 2023 07:17:26 -0700 (PDT) Date: Tue, 12 Sep 2023 14:15:56 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3325; i=ardb@kernel.org; h=from:subject; bh=NjETdn9/OJSsYzpCnaqf4zu6JV6mU8+8GZRaOpY0ZAI=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaM69eIdXF/2mPzzcdnrJPONpJzn5+JY/FuiLSvsdz 1oioZLXUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACZyhpfhf8mKddoBH7umt5y8 pML3qP5WqaHXm1Wi8+Q3euZI+z8/pMHwT1/8+uZGD8FIzfjpbocW/KmVSk/xE8nkexd+sWVZL+9 2fgA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-69-ardb@google.com> Subject: [PATCH v4 06/61] arm64: ptdump: Allow all region boundaries to be defined at boot time From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071729_185325_31B69797 X-CRM114-Status: GOOD ( 15.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Rework the way the address_markers array is populated so that we can tolerate values that are not compile time constants generally, rather than keeping track manually of the array indexes in question, and poking new values into them manually. This will be needed for VMALLOC_END, which will cease to be a compile time constant after a subsequent patch. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/ptdump.c | 54 ++++++++------------ 1 file changed, 22 insertions(+), 32 deletions(-) diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 3958b008f908..bfc307890344 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -26,34 +26,6 @@ #include -enum address_markers_idx { - PAGE_OFFSET_NR = 0, - PAGE_END_NR, -#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) - KASAN_START_NR, -#endif -}; - -static struct addr_marker address_markers[] = { - { PAGE_OFFSET, "Linear Mapping start" }, - { 0 /* PAGE_END */, "Linear Mapping end" }, -#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) - { 0 /* KASAN_SHADOW_START */, "Kasan shadow start" }, - { KASAN_SHADOW_END, "Kasan shadow end" }, -#endif - { MODULES_VADDR, "Modules start" }, - { MODULES_END, "Modules end" }, - { VMALLOC_START, "vmalloc() area" }, - { VMALLOC_END, "vmalloc() end" }, - { VMEMMAP_START, "vmemmap start" }, - { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, - { PCI_IO_START, "PCI I/O start" }, - { PCI_IO_END, "PCI I/O end" }, - { FIXADDR_TOT_START, "Fixmap start" }, - { FIXADDR_TOP, "Fixmap end" }, - { -1, NULL }, -}; - #define pt_dump_seq_printf(m, fmt, args...) \ ({ \ if (m) \ @@ -339,9 +311,8 @@ static void __init ptdump_initialize(void) pg_level[i].mask |= pg_level[i].bits[j].mask; } -static struct ptdump_info kernel_ptdump_info = { +static struct ptdump_info kernel_ptdump_info __ro_after_init = { .mm = &init_mm, - .markers = address_markers, .base_addr = PAGE_OFFSET, }; @@ -375,10 +346,29 @@ void ptdump_check_wx(void) static int __init ptdump_init(void) { - address_markers[PAGE_END_NR].start_address = PAGE_END; + struct addr_marker m[] = { + { PAGE_OFFSET, "Linear Mapping start" }, + { PAGE_END, "Linear Mapping end" }, #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) - address_markers[KASAN_START_NR].start_address = KASAN_SHADOW_START; + { KASAN_SHADOW_START, "Kasan shadow start" }, + { KASAN_SHADOW_END, "Kasan shadow end" }, #endif + { MODULES_VADDR, "Modules start" }, + { MODULES_END, "Modules end" }, + { VMALLOC_START, "vmalloc() area" }, + { VMALLOC_END, "vmalloc() end" }, + { VMEMMAP_START, "vmemmap start" }, + { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, + { PCI_IO_START, "PCI I/O start" }, + { PCI_IO_END, "PCI I/O end" }, + { FIXADDR_TOT_START, "Fixmap start" }, + { FIXADDR_TOP, "Fixmap end" }, + { -1, NULL }, + }; + static struct addr_marker address_markers[ARRAY_SIZE(m)] __ro_after_init; + + kernel_ptdump_info.markers = memcpy(address_markers, m, sizeof(m)); + ptdump_initialize(); ptdump_debugfs_register(&kernel_ptdump_info, "kernel_page_tables"); return 0; From patchwork Tue Sep 12 14:15:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381764 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9369ECA0EEC for ; Tue, 12 Sep 2023 14:18:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=BAMo/PI4ShpLsiNX3OEiVGxLUu3IWjUFBYaxLdJE6EI=; b=sdaRQ4krnusLgyPn2YbuAhquHy tvKyKk7V3v3UaE9QDhn9wsNCdCnWeFbWUZDT97bOOHDQpRpJvw+K482oQKAWCldoxSq5D+SlTieDc CZzl2yDwvMdbxAPAPY9eicV+e0b1bT5aXPuTIgMchNIIeYRanGnlE1WHSe4FO8GUwTyp1DuioBr9A XszhDmOEPQocHJuC+EXcYnHbiFTFGEzcVW42isd0T2NKSumCckCCfQXjO/h/tPFstPuIMQ7jUJhqH cqgWgfq0PQABMZ41r0KdWhN4kSVQDLVgKg5ceqhbpOrlZ8B+iudgpaGPRzleZlh9ZXvlmRfqq0jZp SKjFR1Sw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Cv-003Vwj-1O; Tue, 12 Sep 2023 14:17:37 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Cp-003Vt5-18 for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:17:32 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-d748b6f1077so5501885276.1 for ; Tue, 12 Sep 2023 07:17:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528249; x=1695133049; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6VtqYQJKrcwrjRRCrFoB+JpauHp+ho8ODSZJfFBdsMg=; b=RyClUXxpXviooctSiXrS5tZAFvbgic7xGcnk7Xg+YMs0+g8PXy0T78Y51/Ei23NatH dkcA+M0bvrk05TMMJi99++0t35HikPx6nc5psC2sw96VZl+1J6vsSYgbeoBMT9t98eW4 Te236uaoOIRbgmcP8L0N/gpLb3BouTW3QiPzHmEBHMwIfYPJMMLeIbwRFgOaD0UCawWD qXvWtaeHKdDbN5TRsoK43QaeIVYofbuzAvDhv2i7lyRkVxxNGPwro5oU55jAgvZN5TaD E2kdhxTYEz6aCdXCbjLtB1aD9ArkNSj7xdHdtkxTqJrXygWzDBG99nEVCnYknRciJrd7 ojxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528249; x=1695133049; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6VtqYQJKrcwrjRRCrFoB+JpauHp+ho8ODSZJfFBdsMg=; b=C1o12bsvcKRSU8cAlTmye77N7yHCCKqFrXtqEy6dcSYgU549QWMNT2ZuKzFW+bNOkI jAu37KwgXSGfMHb1669pWG6Xi3tLSyf5NKzaAzHaQqA22Hu4afBTRT4GTv9ggY3ztrWQ fa1vzDCjotFigszigYtVl1OXJud2TNnJyblrsZMnC0lzZRNWkrYxxAEBCigdVtlrJJr/ 0kZJLh2JO/AaowDCwWZ6cHuAt3syhp9h42JdT5cwtT1sNsLRYkbb7/gjrVqs2UEiIMm5 npSQ7sBaM8+Oq1dyI/PmozYR1i0F4C97YLYlw5SQW2ngJp1BhwHI9JdD5vvgOUP0s7T2 konw== X-Gm-Message-State: AOJu0YwPZlTbHC/a0q4vxzQhtxcV9bCtsAmm3G8pJ9U8TsLpTQyHKVl9 xAjLMAlFW7dtgq3fzwmiRDRRqbZqx5wK1AB3upgjXZDg/1u98LPDfRw7j9DzyS6/vPpeqn12Rpf cDZwErz4G2OczDMdX4SrJuViAUR9oFFnJcM6pcHJcLDoPX4hCN6TQBGWazPahy3cF2ZRTlX9tjU c= X-Google-Smtp-Source: AGHT+IFCoV8rnkqLnDcmMNk8Jtu6Y+OH3KxAteyfyVrdfTMzTH0LefunZU/mX2HyyvfMLD7xeTJ56X54 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:ae96:0:b0:d7e:dff4:b0fe with SMTP id b22-20020a25ae96000000b00d7edff4b0femr258433ybj.7.1694528249269; Tue, 12 Sep 2023 07:17:29 -0700 (PDT) Date: Tue, 12 Sep 2023 14:15:57 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1391; i=ardb@kernel.org; h=from:subject; bh=S76KNRxsYzjbzZ104dYTboAUISXchNmDbYsiRupFZPE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaO6rKM6Ge+GMKgqim4/2nJbPF9Fb5vBWdUlGtUrl+ 47zX7I6SlkYxDgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwEQ29jH80/p7+6lC0GZDs1PL k6cH+jYUXm0/GpjLbhYhl9RxNdmQgeF/9MI5wo1HVGo2uL1nXmrdciFE6nPpEcEjeVnpOo+jdzY zAAA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-70-ardb@google.com> Subject: [PATCH v4 07/61] arm64: ptdump: Discover start of vmemmap region at runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071731_405691_69E9AD03 X-CRM114-Status: GOOD ( 15.15 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We will soon reclaim the part of the vmemmap region that covers VA space that is not addressable by the hardware. To avoid confusion, ensure that the 'vmemmap start' marker points at the start of the region that is actually being used for the struct page array, rather than the start of the region we set aside for it at build time. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/ptdump.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index bfc307890344..f3fdbf3bb6ad 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -346,6 +346,8 @@ void ptdump_check_wx(void) static int __init ptdump_init(void) { + u64 page_offset = _PAGE_OFFSET(vabits_actual); + u64 vmemmap_start = (u64)virt_to_page((void *)page_offset); struct addr_marker m[] = { { PAGE_OFFSET, "Linear Mapping start" }, { PAGE_END, "Linear Mapping end" }, @@ -357,7 +359,7 @@ static int __init ptdump_init(void) { MODULES_END, "Modules end" }, { VMALLOC_START, "vmalloc() area" }, { VMALLOC_END, "vmalloc() end" }, - { VMEMMAP_START, "vmemmap start" }, + { vmemmap_start, "vmemmap start" }, { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, { PCI_IO_START, "PCI I/O start" }, { PCI_IO_END, "PCI I/O end" }, From patchwork Tue Sep 12 14:15:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 23333CA0EEB for ; Tue, 12 Sep 2023 14:18:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Yij5o6xR9s3m/yDAkHdrWK4MdcpJeSpqh9Kb8j+MtLA=; b=hpflPTeAJ8X+qdg0F3yQ7zOEZ1 BmgiXjLUqvT7nzu19byeQItk6pQ2PfDqCxz3hV2kk/bQgigebpL3bRbcaCSjkl+8dEzuk7zpGSbqd d1KtGioctlHcezvK3y8EzOvRncG6viS+l8/N/X4o3fR92tqopQVu9FMuogxZywR7qjZWhI4x8wQpU dHq6L6PLjOdnuMjcYN7jIWzYiefWAkvUcpwBCXqjxVjOrHwdJjxu2ydns9N1u2Pgf6lHB8Vr08iaE 3sylEdiNaqkBPAL9+7obj/EdNJIQn6UqfM4YRMiCW15KtZ+fHojJvyi2iD9tmFMoSc8Ipp6blm4oa 7yx23mBA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Cw-003VxK-02; Tue, 12 Sep 2023 14:17:38 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Ct-003VuS-04 for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:17:36 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-4011fa32e99so43404805e9.0 for ; Tue, 12 Sep 2023 07:17:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528252; x=1695133052; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZdYJARCA6Ht1ut9zO/LmlVjbDWZjjlgcF9vUFvkWOlY=; b=2a+C2vqH/BYJUZTguhtFfkrFyHE1N0jSkS72ASZNpeLc6f3449L6je2YealZqF4zYe VxmeUlMrKtVO/Znx9iFZh9FkSTo50bAJVJgFO/8kiK2/QnndHM6gnggnTTCz9FQpkHhG JTvNFejHemY1pWjEVNzTJdJYdBUoUZdQmF/d+hqNOXtFBMQ1pieQlf7bblT/a+ti2UJI 88yThvfVFyHdzIFMev32W5mfAqmbJ0XK5WcgEqCWv9HMlWADYhK+pDmwT8isSIf0LICV yVZhDVDL7WMuKkwIpJaY96bRN9iDV/V6drQUZeyXMdmEcBlDsyaK5R8WbCh8T4FaCVYx 7qoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528252; x=1695133052; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZdYJARCA6Ht1ut9zO/LmlVjbDWZjjlgcF9vUFvkWOlY=; b=Vd45qEWyKeAmG7Bw+TYL9AQpPuC9t5ewwltPEGY4XbXU65UE7p1DRc85QNdNM4RGJd 6QdRux9HqzDoDQN2HxjonBBGUoV3IHIveF4fEEYpoxxKvcU9FI/GhCjvdLk4WF0a2Neg pHE15nvi/LnxCSwBDSSaPr+KPyJfwmPE48Ye3XrDwLWA0gkcYk0W7c15jowLPMSJ9dqU lzA2EPFHvMstgaHdDxMbuYeTnunW1Om1WxzF+PAzpcnHpQlR5UdWm2BL+C6ApkLWrCEh hO7UbKdIq2JXoT5UsrZtGz7nU4Ncny6sL0DWJS/Q7TF+CIfRJEtgBDcpUP01vmdMm9pH fjpA== X-Gm-Message-State: AOJu0YxhzuWim0M1gpLmm4jrdqo3G5L0kJCMwZg78wuKbE8eh9I1vS8E soRC1JemjJZX+9lgm502k5YAK16YIv13aUVDA5s8Ulb5p7+e2tWWwhqG/wCQQ/ApxHis6TwF4ye lH/pdh1ZB8a3jkjhHGp1ZBWAS17cW9v1J+MbYs2rT+TO8K2vQLJA+J8YBzbzyz1BTCZTWzuFmLA Y= X-Google-Smtp-Source: AGHT+IGWoaGUfuIY3ZYrtgIqh56Lv6Gfzqd/4LwiWV7EkoJPjO3kc76BOS1OjtyLJc5CnP+b4MIUF1Mi X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:f64c:0:b0:31a:ed75:75d5 with SMTP id x12-20020adff64c000000b0031aed7575d5mr147856wrp.6.1694528252542; Tue, 12 Sep 2023 07:17:32 -0700 (PDT) Date: Tue, 12 Sep 2023 14:15:58 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2456; i=ardb@kernel.org; h=from:subject; bh=fXe/v+wKmpChZFWr3ArtHIQDI4gcTDLCPu1fk5vCIYA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaF6QDetXDqbtf+fddVDLvn6sgClhp57CBe21vf+ap 93o6DDpKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABORusTwP91u4WPOOMd1Zi9W MvlfDvsjlK2jedT1bU2GzvoG/vm58xgZrul9Y1CYmvvgzoTEV+ZrFzspHI1aU/Pm9dwJOd92W20 LZwMA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-71-ardb@google.com> Subject: [PATCH v4 08/61] arm64: vmemmap: Avoid base2 order of struct page size to dimension region From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071735_058417_756BA481 X-CRM114-Status: GOOD ( 17.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The placement and size of the vmemmap region in the kernel virtual address space is currently derived from the base2 order of the size of a struct page. This makes for nicely aligned constants with lots of leading 0xf and trailing 0x0 digits, but given that the actual struct pages are indexed as an ordinary array, this resulting region is severely overdimensioned when the size of a struct page is just over a power of 2. This doesn't matter today, but once we enable 52-bit virtual addressing for 4k pages configurations, the vmemmap region may take up almost half of the upper VA region with the current struct page upper bound at 64 bytes. And once we enable KMSAN or other features that push the size of a struct page over 64 bytes, we will run out of VMALLOC space entirely. So instead, let's derive the region size from the actual size of a struct page, and place the entire region 1 GB from the top of the VA space, where it still doesn't share any lower level translation table entries with the fixmap. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/memory.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 2745bed8ae5b..b49575a92afc 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -30,8 +30,8 @@ * keep a constant PAGE_OFFSET and "fallback" to using the higher end * of the VMEMMAP where 52-bit support is not available in hardware. */ -#define VMEMMAP_SHIFT (PAGE_SHIFT - STRUCT_PAGE_MAX_SHIFT) -#define VMEMMAP_SIZE ((_PAGE_END(VA_BITS_MIN) - PAGE_OFFSET) >> VMEMMAP_SHIFT) +#define VMEMMAP_RANGE (_PAGE_END(VA_BITS_MIN) - PAGE_OFFSET) +#define VMEMMAP_SIZE ((VMEMMAP_RANGE >> PAGE_SHIFT) * sizeof(struct page)) /* * PAGE_OFFSET - the virtual address of the start of the linear map, at the @@ -47,8 +47,8 @@ #define MODULES_END (MODULES_VADDR + MODULES_VSIZE) #define MODULES_VADDR (_PAGE_END(VA_BITS_MIN)) #define MODULES_VSIZE (SZ_2G) -#define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT))) -#define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) +#define VMEMMAP_START (VMEMMAP_END - VMEMMAP_SIZE) +#define VMEMMAP_END (-UL(SZ_1G)) #define PCI_IO_START (VMEMMAP_END + SZ_8M) #define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE) #define FIXADDR_TOP (-UL(SZ_8M)) From patchwork Tue Sep 12 14:15:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381766 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E2807CA0EEC for ; Tue, 12 Sep 2023 14:18:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=JjcZB9uLg4uiecvWiSxQSiDj4MGYX7l7JVBqeQQ6O+o=; b=Jdn8M225XwyTL7RzDX7WkDwLkl 1ZebRRjrkczd2T7/tnjq4WXshl/JclYTjk4Bq2HC5WMaAujAVAEbPfLPC/r2eE6pkW92mRDRF9ukk BP8W1KMx7DerUD9Y+VO/aH+jPCpJFgmpdsfsOtADX/QFA2Q/cXAgZF/W7RwFKMnNCzgPDXP1zevg5 zzV5hg/hnHcmAIiwRZTyyOojIjSeGN1s3rQrZbdNH1v8hW1MRPoOvkV1nvvKWkXMRngtHVpYEgPFB UKu/c0ZSaHp+UDsUih1xSa3nDkb+BGuJ3Vzk+ObaKx6c/WoDqwh2DpjlAWnKbUW3j2d4oYDONVwC1 poRzKBjw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4D2-003W2a-2A; Tue, 12 Sep 2023 14:17:44 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Cv-003Vw1-1N for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:17:38 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-4011fa32e99so43405315e9.0 for ; Tue, 12 Sep 2023 07:17:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528255; x=1695133055; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KpTkuU4FZ8Pg9y3MG1ea5g1EA9hSfn/J6adbhCYXU2U=; b=AlcwQC9D3vPuXtCAacMRwAD9zRGmmVao0HiQgfDNp+qdYvCo9ukCtrGZpbGeIwqZKN JJEOXwDgBP6YoBMktA7mAWLmxRU4XqSTK948GVWpmxcRJFt17pJ4qR7t0w3CnJPuQ/J2 hdqNY1awvjS2SOX1OgcJEQTpIJqiM9rD8x1A6wtlePtqieZk7531UwZxR+0n3m5G6ux9 IKAr7s6XTyAkkAZD9cftSaKxnnbbyiXBA1DmrtyA5B9Hja3j/xG/7jWKGcf9m9qlB7+i yaM6fd8LKJoErghIdKcGN7/hsjA9pmoQ1d1iwsaSxcFdXlYOs1t7+RwmpFJ4vLVeldAk 8gkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528255; x=1695133055; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KpTkuU4FZ8Pg9y3MG1ea5g1EA9hSfn/J6adbhCYXU2U=; b=rFbphJQZl5bFw1pva1wqJa3jtiuAoL8VtPMYlUL86h6eNHodf8WlZoIFrSPxOcKHqk vaNQmqX4In1C1UXHjFD/L3XYZ4Wj7Sq8eCi7WEJt46KfJbOvILYPqaRNkHSN0dwzWj/8 reGWDxDkId5OWWgLdXBswsX6Zf+kdapPur9Hky0q6T2gv+LrvlaASPyaS0eNTK/osVwE uytIVHbhHmZOowH+TdkaEpuPnM5y0W4hZrXMLtYWMOMEadXPXIX6n0B8RdR7q+1vhV+D jRiGRY8sFEP1plbHDJ8+0gYY7GfcPiC43MRtGO0YhaxnP+nFTV1VKscnZoeQuiv8u3wH YQ7g== X-Gm-Message-State: AOJu0YwB9aHpj7e1DqOeawdpuz24X7z+03RTTHWr67ix62dqT439PzkM VAWZaWER9I+3aHlXQB2fm3e9TEBNIGFX6iXYqYx2i2IPERj9bOQ18U2p8OGZCBvTPsxKk8vZn4a HlBNYnCq1iBYbpwjSQ+uBLSgo2Qc2oR5wExMNdfTPAyHOdVLM3hzp8m0J6jFte1UKNRwS5w0UAn w= X-Google-Smtp-Source: AGHT+IGRcYlprdoWqBYlqJugfRGHwsYhjNCDUoWvSihdK+MrW1unOYWKIV0njV28ELXzzxWNccf/tNJn X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:630e:0:b0:317:5169:66ec with SMTP id i14-20020a5d630e000000b00317516966ecmr155095wru.14.1694528255183; Tue, 12 Sep 2023 07:17:35 -0700 (PDT) Date: Tue, 12 Sep 2023 14:15:59 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1790; i=ardb@kernel.org; h=from:subject; bh=Z86mK7KgtSrSmoFooFJ6pjc/HBeI+NJWVcvgU20xk4k=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaL6EafDlzb8q5FWYl8m8Ohls/v+TkLY105O3/6flv V7aeFCgo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEyk7x0jw/U/BZWhAjOYD2Ve 2XeF8YORZPDOHD/XPWrWUp82LRX1/MDI8PPojcOCAU1v8+/fSZYKPnrvzeYZyQlr8xlPzWyr7vA L4wIA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-72-ardb@google.com> Subject: [PATCH v4 09/61] arm64: mm: Reclaim unused vmemmap region for vmalloc use From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071737_470447_60F8632E X-CRM114-Status: GOOD ( 15.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The vmemmap array is statically sized based on the maximum supported size of the virtual address space, but it is located inside the upper VA region, which is statically sized based on the *minimum* supported size of the VA space. This doesn't matter much when using 64k pages, which is the only configuration that currently supports 52-bit virtual addressing. However, upcoming LPA2 support will change this picture somewhat, as in that case, the vmemmap array will take up more than 25% of the upper VA region when using 4k pages. Given that most of this space is never used when running on a system that does not support 52-bit virtual addressing, let's reclaim the unused vmemmap area in that case. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/pgtable.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 01fddcc2ae4d..1858abd51b8f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -18,11 +18,15 @@ * VMALLOC range. * * VMALLOC_START: beginning of the kernel vmalloc space - * VMALLOC_END: extends to the available space below vmemmap, PCI I/O space - * and fixed mappings + * VMALLOC_END: extends to the available space below vmemmap */ #define VMALLOC_START (MODULES_END) +#if VA_BITS == VA_BITS_MIN #define VMALLOC_END (VMEMMAP_START - SZ_8M) +#else +#define VMEMMAP_UNUSED_NPAGES ((_PAGE_OFFSET(vabits_actual) - PAGE_OFFSET) >> PAGE_SHIFT) +#define VMALLOC_END (VMEMMAP_START + VMEMMAP_UNUSED_NPAGES * sizeof(struct page) - SZ_8M) +#endif #define vmemmap ((struct page *)VMEMMAP_START - (memstart_addr >> PAGE_SHIFT)) From patchwork Tue Sep 12 14:16:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381765 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4D8ABCA0EEB for ; Tue, 12 Sep 2023 14:18:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=S556F9/ZlVVRLnLZSf6vOBH1mD4mhxpG45u7JnTEA+Q=; b=vLzbIFzj+/I8HjdehjOahabi9q FqzwiEX8cuPyL3fEENhSuzjv5MqIzGGiWnkBL8Qkk46iwbFwClNcJ+CW3TY/UZpQ0naTvDIFOjsXH 1zGEEsVsovVPeH7z+hzyRMs9GCk1kbRPc7Dlvmr8AYwQoMcOwZwSAsBRGGRw2FOoTJWX+LrK+Bi42 +18O8dXM2+b0TqXk3k7BCelJet7U82ECuA9bNIC9+ZjGKNIYeyoBWN7MQF1MQf5fvVUzdiln79moj mKP3osl2hPZ5CG+Cjqyfj5AyT686ORLU7MFlv9qEcm7c4D2Iwir7dAXBLOtbwQndQfFeTigNWVMjQ tmZgmFzg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4D3-003W3I-2o; Tue, 12 Sep 2023 14:17:45 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Cx-003VxL-0E for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:17:40 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-58c8cbf0a0dso122419657b3.1 for ; Tue, 12 Sep 2023 07:17:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528257; x=1695133057; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ONCaJLYcCgdni3j1wgWy9nMUxs3EQ633sZ++p1ZZL0w=; b=pb4c7sgkGkp5IDXcvEnIij/FLw+KWess2zeimRxKH2KIzbAMyJM5jl/BfF2MVhWqh5 j3aO4LK76De4V6bOFSytz6urj8MT7Rd1l5jXVdrGvBzRwkEZGfeFeZRViaP6x4qz2sa3 VuTeje7zNaqr1jWAr1X5n/hxsbtCVVUKau2a7vmPVawQqYbylWNsuMjlH/YmKvm/2c6l fYl5+PpN/IiT5uZBPShetUapZFBbLmVHVbyliM4Yz3B9TmcfptI0jwOnP91LITTp5f1F b/MHUtOPSnWUZxA0dPRn6hxwUWrJSOjtTGu1xN+17/+V7A9AjqU2bcWO55BiHE9gLllm 1Smg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528257; x=1695133057; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ONCaJLYcCgdni3j1wgWy9nMUxs3EQ633sZ++p1ZZL0w=; b=wlIfVRCxF/jQfHts1pSSC98BjTNIc/t+95oQ8ez7T9aNVYju/GWTEMgXs/CW1eahj+ WIjcPLzTsibF23/YLpwDs/GFWTVL0X37sXhQQg3YLseeD/PExe9Ii7jfSW98kjOxH6a7 YaagPQ1WGkfr2erfXwUx36sI44vjrTxdOrOM6E//6LOz1JZN5NweqTDCnL/dmZQAvCFB wqNehmQirCYU+vdMQ4dxU53C1TOUtoSD9/gRvPtjYDFWdpdYTPwZjOpcLZHoFqUFJ7br M/Iy+BHY6jqhZpfqDATDMOx3XXgpwUOn+8AAG/mtCgWUWHvwzZ505quqwbGQKQx1M/rj QJRA== X-Gm-Message-State: AOJu0YwgylYPC5/Ad1l4SxIl281kUfb5lI9ylyD9jzY3uIz6yvVtnFBa P8WVzd4PblVbdRu+HNsJiDxs115WU6Hj77LmLBSJfQ72777YIZbBqj16vP2ilWgxdICQbmCgVc7 msLRvyQ2kXeIhpQELZ5v6S2JDCdSW3jQzE1jegLpVMPUYifH0bmKssBDZav10rnCiHw5k0Oxssx c= X-Google-Smtp-Source: AGHT+IE4eMaEjl+bOnVc0xBss16iPzVzTmjCOvT3C/l5PX3Tf5uznrV6XLBewZyPqVR+aiGw3djMhCle X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:bc4a:0:b0:59b:b0b1:d75a with SMTP id b10-20020a81bc4a000000b0059bb0b1d75amr32621ywl.4.1694528257636; Tue, 12 Sep 2023 07:17:37 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:00 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2763; i=ardb@kernel.org; h=from:subject; bh=DdRQ4AAoBXXFaR1Awn7Do8PULwk4CqCK4XGt9V+4C0c=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaEElz6u/tdGndgSm15+yk5ebJLH0qe13q9r5HVGG0 21X1ER1lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgIm43WBkuHBpXs+kToGP1oXs M+M839/9LRiQWhUtmRJ5gMe/99qzYwy/mI6FG8xiP9XrO3sxY4SpoMTS+73xi04GXrfW6lxmlFH ODAA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-73-ardb@google.com> Subject: [PATCH v4 10/61] arm64: kaslr: Adjust randomization range dynamically From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071739_119278_63AF2CEA X-CRM114-Status: GOOD ( 16.86 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Currently, we base the KASLR randomization range on a rough estimate of the available space in the upper VA region: the lower 1/4th has the module region and the upper 1/4th has the fixmap, vmemmap and PCI I/O ranges, and so we pick a random location in the remaining space in the middle. Once we enable support for 5-level paging with 4k pages, this no longer works: the vmemmap region, being dimensioned to cover a 52-bit linear region, takes up so much space in the upper VA region (the size of which is based on a 48-bit VA space for compatibility with non-LVA hardware) that the region above the vmalloc region takes up more than a quarter of the available space. So instead of a heuristic, let's derive the randomization range from the actual boundaries of the vmalloc region. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/image-vars.h | 2 ++ arch/arm64/kernel/pi/kaslr_early.c | 11 ++++++----- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 35f3c7959513..2cc3aa4c27c7 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -34,6 +34,8 @@ PROVIDE(__pi___memcpy = __pi_memcpy); PROVIDE(__pi___memmove = __pi_memmove); PROVIDE(__pi___memset = __pi_memset); +PROVIDE(__pi_vabits_actual = vabits_actual); + #ifdef CONFIG_KVM /* diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index 17bff6e399e4..b9e0bb4bc6a9 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -14,6 +14,7 @@ #include #include +#include /* taken from lib/string.c */ static char *__strstr(const char *s1, const char *s2) @@ -87,7 +88,7 @@ static u64 get_kaslr_seed(void *fdt) asmlinkage u64 kaslr_early_init(void *fdt) { - u64 seed; + u64 seed, range; if (is_kaslr_disabled_cmdline(fdt)) return 0; @@ -102,9 +103,9 @@ asmlinkage u64 kaslr_early_init(void *fdt) /* * OK, so we are proceeding with KASLR enabled. Calculate a suitable * kernel image offset from the seed. Let's place the kernel in the - * middle half of the VMALLOC area (VA_BITS_MIN - 2), and stay clear of - * the lower and upper quarters to avoid colliding with other - * allocations. + * 'middle' half of the VMALLOC area, and stay clear of the lower and + * upper quarters to avoid colliding with other allocations. */ - return BIT(VA_BITS_MIN - 3) + (seed & GENMASK(VA_BITS_MIN - 3, 0)); + range = (VMALLOC_END - KIMAGE_VADDR) / 2; + return range / 2 + (((__uint128_t)range * seed) >> 64); } From patchwork Tue Sep 12 14:16:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381767 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7AD58CA0EED for ; Tue, 12 Sep 2023 14:18:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=zCFi5lAmo9wOYofmVjS4Z5TK4pzBMGC+SH3lIDO1xMI=; b=uDgKfQXQNsjwlwWjn7jvKPX1Xe R2vWtBis/1yt4LV9QUXLmz8xXvkGKbO6Zkp3RgRLqxTm+yrHrxSqvyZeChkkfUFBRjYBm0C+Gt5qE LiEZGoMZFQh+G4sc+JYMNerx8r255AunFu0rXs77dgRzbYG8M4ibR2v5pN1i3eEKfpvPCXtxERGHQ C8kviwFLU214rjJgNDq7mY+cc47dO79vxbhjkRiCa9P0lvfTOpCRZIpDiKkiBOL/TsjkLMQyRi4IW cHhXBCwlAqASw3O7FQSWooJan4bcJx/17CgwZlMPUkebrZFrlX4QhvsO6fDGMyK9s2Yin2ZutCBmy wXQvOYeg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4D4-003W4R-2r; Tue, 12 Sep 2023 14:17:46 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Cz-003VzN-0p for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:17:42 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-59b5d4a8242so42911807b3.0 for ; Tue, 12 Sep 2023 07:17:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528260; x=1695133060; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=JMOcmR6gwIMeL1pByZVoCRpSMNbE6fe+QeZKiWAXH4c=; b=eHFKT0aCkJ7GnuCWdYW1Kz3RS8lfO0nsEYo7dKhcrHcO+GVBp3wFvAHawTDmhixGfr We+7m+roG8JWLi9c77m5ingc1MVMHGGJtnEbOetPkOPjKbQWFVqsUy07sR9UzUC+JpkE S51WtUKrc8OFpCHW0le/9vleb4ScjEGvoKdRuN9Lb7MrCdZzlYqXdcposGdZ8yRUvsFX YnlUfknkYUhCGhq09iNZKbwL+GkUI70C62QO2EBX+rq4LtblMvS7ldGCpHa5NwqKQ8w1 uhDz+/fRq91uWnM9GFdmHhw7MwoqlD1vmsMoDEQFhy0RtWZVeLyLO3YYcj49Feh0GqUM f16Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528260; x=1695133060; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JMOcmR6gwIMeL1pByZVoCRpSMNbE6fe+QeZKiWAXH4c=; b=a1ldKV9/qx46EKurWHlh43mgdCm+IfkDWzWg78+m45G+VQ6ZzvKxIdSjizN4Ag/tYH GTBvXHTFHdlA4w5R6PM4n+ndYw3sztQ4BMDXZOQn1xqps/UyHmN+l7/wlUeBPhO0oCWC qpW87xQdKJCwoczXoFEvr62wWZa8n8Gp26bIC4gaWXBIUerD4wnv0KG34nI1BEj1iakB YDTOZbfQ7SnnC7CaU/dg7NIXMesVbXMRU3Ux1aeldxzRuElKBv9UOCTGSmR8rJWnBhEl SMLPK7wvVDL5v4XkwwpwD7QGjIR0Z1UQP89tmcpls4o822Iy5IuZ+S2ctlLEJ/RFFqkR 1E5w== X-Gm-Message-State: AOJu0YywSDKGaXuNNzNCDoaKhs4F7zyXnYRmSG4BcHOo2QrzhA8IHLtp eJoIWHXYiPR0l4b3sfIjvTXUA954HIi65zQcEG8WTJc2oDA3W6nMznuZEzPxCVoMaeMU8r9FNHY LdVj+WdEvEUzASWCt+FSVL2E18AtDjWetNURhNjfgMrQRxyOat1oJJQKERI/AGB/3fzGxAZ35F2 k= X-Google-Smtp-Source: AGHT+IGRKDxpjX4m8db72UgMkSwt0UdKqFM/3cTTk2G/ALaKkvS7/R+PGgLoAQPQ3nj5vhxDWkBl9UUk X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:ad04:0:b0:584:41a6:6cd8 with SMTP id l4-20020a81ad04000000b0058441a66cd8mr299236ywh.8.1694528260131; Tue, 12 Sep 2023 07:17:40 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:01 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6463; i=ardb@kernel.org; h=from:subject; bh=HYrF1i309+7W6RY+1XcsEkqxhtaRh+j6zQpK8dqLGDQ=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaOFZf82Um+lTVaYeudPufWkZZ/xpw84o59Ii3mt5i 61Ej2/uKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABOxLWRkeMt1+JTkvge/ReR7 zr3u+PLZZvWzrfME1c6z3nEsPHnQN5fhryyXdYStW6+dQ7JH0+1jX9cLRP7++PvH30xzIaElc75 ZsgMA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-74-ardb@google.com> Subject: [PATCH v4 11/61] arm64: kernel: Manage absolute relocations in code built under pi/ From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071741_301466_DF8150B3 X-CRM114-Status: GOOD ( 25.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The mini C runtime runs before relocations are processed, and so it cannot rely on statically initialized pointer variables. Add a check to ensure that such code does not get introduced by accident, by going over the relocations in each object, identifying the ones that operate on data sections that are part of the executable image, and raising an error if any relocations of type R_AARCH64_ABS64 exist. Note that such relocations are permitted in other places (e.g., debug sections) and will never occur in compiler generated code sections when using the small code model, so only check sections that have SHF_ALLOC set and SHF_EXECINSTR cleared. To accommodate cases where statically initialized symbol references are unavoidable, introduce a special case for ELF input data sections that have ".rodata.prel64" in their names, and in these cases, instead of rejecting any encountered ABS64 relocations, convert them into PREL64 relocations, which don't require any runtime fixups. Note that the code in question must still be modified to deal with this, as it needs to convert the 64-bit signed offsets into absolute addresses before use. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/pi/Makefile | 9 +- arch/arm64/kernel/pi/pi.h | 14 +++ arch/arm64/kernel/pi/relacheck.c | 130 ++++++++++++++++++++ 3 files changed, 151 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index c844a0546d7f..bc32a431fe35 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -22,11 +22,16 @@ KCSAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n +hostprogs := relacheck + +quiet_cmd_piobjcopy = $(quiet_cmd_objcopy) + cmd_piobjcopy = $(cmd_objcopy) && $(obj)/relacheck $(@) $(<) + $(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ \ --remove-section=.note.gnu.property \ --prefix-alloc-sections=.init -$(obj)/%.pi.o: $(obj)/%.o FORCE - $(call if_changed,objcopy) +$(obj)/%.pi.o: $(obj)/%.o $(obj)/relacheck FORCE + $(call if_changed,piobjcopy) $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h new file mode 100644 index 000000000000..f455ad385976 --- /dev/null +++ b/arch/arm64/kernel/pi/pi.h @@ -0,0 +1,14 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Author: Ard Biesheuvel + +#define __prel64_initconst __section(".init.rodata.prel64") + +typedef volatile signed long prel64_t; + +static inline void *prel64_to_pointer(const prel64_t *offset) +{ + if (!*offset) + return NULL; + return (void *)offset + *offset; +} diff --git a/arch/arm64/kernel/pi/relacheck.c b/arch/arm64/kernel/pi/relacheck.c new file mode 100644 index 000000000000..b0cd4d0d275b --- /dev/null +++ b/arch/arm64/kernel/pi/relacheck.c @@ -0,0 +1,130 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2023 - Google LLC + * Author: Ard Biesheuvel + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ +#define HOST_ORDER ELFDATA2LSB +#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__ +#define HOST_ORDER ELFDATA2MSB +#endif + +static Elf64_Ehdr *ehdr; +static Elf64_Shdr *shdr; +static const char *strtab; +static bool swap; + +static uint64_t swab_elfxword(uint64_t val) +{ + return swap ? __builtin_bswap64(val) : val; +} + +static uint32_t swab_elfword(uint32_t val) +{ + return swap ? __builtin_bswap32(val) : val; +} + +static uint16_t swab_elfhword(uint16_t val) +{ + return swap ? __builtin_bswap16(val) : val; +} + +int main(int argc, char *argv[]) +{ + struct stat stat; + int fd, ret; + + if (argc < 3) { + fprintf(stderr, "file arguments missing\n"); + exit(EXIT_FAILURE); + } + + fd = open(argv[1], O_RDWR); + if (fd < 0) { + fprintf(stderr, "failed to open %s\n", argv[1]); + exit(EXIT_FAILURE); + } + + ret = fstat(fd, &stat); + if (ret < 0) { + fprintf(stderr, "failed to stat() %s\n", argv[1]); + exit(EXIT_FAILURE); + } + + ehdr = mmap(0, stat.st_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + if (ehdr == MAP_FAILED) { + fprintf(stderr, "failed to mmap() %s\n", argv[1]); + exit(EXIT_FAILURE); + } + + swap = ehdr->e_ident[EI_DATA] != HOST_ORDER; + shdr = (void *)ehdr + swab_elfxword(ehdr->e_shoff); + strtab = (void *)ehdr + + swab_elfxword(shdr[swab_elfhword(ehdr->e_shstrndx)].sh_offset); + + for (int i = 0; i < swab_elfhword(ehdr->e_shnum); i++) { + unsigned long info, flags; + bool prel64 = false; + Elf64_Rela *rela; + int numrela; + + if (swab_elfword(shdr[i].sh_type) != SHT_RELA) + continue; + + /* only consider RELA sections operating on data */ + info = swab_elfword(shdr[i].sh_info); + flags = swab_elfxword(shdr[info].sh_flags); + if ((flags & (SHF_ALLOC | SHF_EXECINSTR)) != SHF_ALLOC) + continue; + + /* + * We generally don't permit ABS64 relocations in the code that + * runs before relocation processing occurs. If statically + * initialized absolute symbol references are unavoidable, they + * may be emitted into a *.rodata.prel64 section and they will + * be converted to place-relative 64-bit references. This + * requires special handling in the referring code. + */ + if (strstr(strtab + swab_elfword(shdr[info].sh_name), + ".rodata.prel64")) { + prel64 = true; + } + + rela = (void *)ehdr + swab_elfxword(shdr[i].sh_offset); + numrela = swab_elfxword(shdr[i].sh_size) / sizeof(*rela); + + for (int j = 0; j < numrela; j++) { + uint64_t info = swab_elfxword(rela[j].r_info); + + if (ELF64_R_TYPE(info) != R_AARCH64_ABS64) + continue; + + if (prel64) { + /* convert ABS64 into PREL64 */ + info ^= R_AARCH64_ABS64 ^ R_AARCH64_PREL64; + rela[j].r_info = swab_elfxword(info); + } else { + fprintf(stderr, + "Unexpected absolute relocations detected in %s\n", + argv[2]); + close(fd); + unlink(argv[1]); + exit(EXIT_FAILURE); + } + } + } + close(fd); + return 0; +} From patchwork Tue Sep 12 14:16:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381769 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A17E7CA0EEC for ; Tue, 12 Sep 2023 14:18:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=AQV2rrDJDj4hB9qW0btijphhe92naA1GW5+VRb02Vtk=; b=zRfNUTSeLUnYKfcJafNEz+39bI RJIn81cuG1B8cFTevEJ8IZqXe/7xYvCBZnJVwwnFftqyQSdlRWIdH9FzcAlRYcDDgO/6XGoRS2Nzo eMsGXPDBHbbVbuneCHo5rpVB6bapFNRwiFg1RYOjVYnc4SZYuotS1Ak2RyQtcuXTT7RWfDNp3b6Mm 8ZiupDz+oWvHwds4N51aHLMmPAZk1Za53QzA81BXssMnSI+KstG+noV2heGCGvvhKVcoQBR2eJOAx l1jBjRpIf6GOpbuTicA/v/ypXiufaYt79vyNZMI08vkU8yQ7vXfpjrhhQMl73GkfQ7U02S1ad7ocP A7RwKH0Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DB-003WAn-28; Tue, 12 Sep 2023 14:17:53 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4D2-003W1P-1p for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:17:46 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-31aed15ce6fso3513811f8f.3 for ; Tue, 12 Sep 2023 07:17:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528262; x=1695133062; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=o1p8QhGrSOJnEfRnJgO3Gr82D/HqkpvvrjBKrSUU2qw=; b=YcztDWphP/FvXXEg138Zvyc7Ugur+/63WgzIm/ro1ErKyTYkhMAkyqSj56s0zi9vzb 0Th6PYWQX1opCGxd/tivICm4bzjS6CHec3RXiro8Zqf23Y40a4y7NYoixh/xYUWIfHHc zuhnDbohPhfp9ZCZmHS2cK9WL/zcc727ZgIyxadsShcxmYOT4/IGTgGJbXm/Tk49x3fL T2YKyvooXJK47UcOXBs1vHCX+1vJ2Nbw4FJkk6CCQ7G9Y9CzujguXqKGCgSmQLvYs6xI Yoqt6jCi/tQAGLTROfZ3UEC7UaCeRxP5DQkZ3K32lo7CjgWup+I79AQwcyUXRhDsEtYW PKSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528262; x=1695133062; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=o1p8QhGrSOJnEfRnJgO3Gr82D/HqkpvvrjBKrSUU2qw=; b=IZeiAUn/62dcp+dsSw9Cw8Y++KOJ3k7oKbq9nmSkFQv0QDzlc8Bj4ECS/X/R2txNgv 5mmQd5DqNUJ0g7ikkc9Ih9iUkEadDfCzUxmMR3CHgx4O4709ADqLkDa/EAL1+0//QcSR 85ujLLaEhwL1bYaFaWDExbolTUPVWtfJO3UfjF/87sO3AzmeK1w81TbguP59B/j6I88Q lHLffCbdSrlbOS0eJ2SQHgcXR4FGc58bE2yWYhF2G0TYujFs/fkBzK+J1BqURGKNTaER QpELS/rXEEH6N4jU0DLRK4O2XIrUGy/YMmXRtZEITLNaDIG6SgXYyfGKbH4C/aeSKRpU FH7A== X-Gm-Message-State: AOJu0Yyjt5VIlZhe2RaHhH5VZhphRlXpXkMJHtIwWJ6zacPvnbPb/Z9t L4fOeY4663TP4VgZvTpC7fgtsDt7zDR/VG+Fip00ubdxVwdRrEkzORb7rBn9YQ4bRIQjffS3jMQ Da3U3nwoxm12aE9OLr3XwNGlzWDmpPh/OItupdH8d0sQUNt/yCNWbbRsGbUsLgSbKptmJJayCJR w= X-Google-Smtp-Source: AGHT+IFC9cURejZGHuevHcnt9rNBSPpIiR6XNZjLPPi6vHSbOh8NXylIu3BWwb88GBbhdzZJBQnr+NXa X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:156b:b0:31f:a664:f871 with SMTP id 11-20020a056000156b00b0031fa664f871mr85937wrz.9.1694528262421; Tue, 12 Sep 2023 07:17:42 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:02 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3211; i=ardb@kernel.org; h=from:subject; bh=mkUUOQwVOlWxqy3A+z2AA0sKLruJdOeHpAY1GKzvtWw=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaNFyY5/AXcdX8jZzakZePqbxevKO4mN8c6fZl1zra FM2+6jQUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACZixMvIsCFbYXFq2EbZWVU/ NrjHT/nonftNT2/CkhseZ5k66kpljBn+aTz/pxRuZPHdP6/xpmJwTOW/uR2C78TqKu/Xt10W7r3 IDAA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-75-ardb@google.com> Subject: [PATCH v4 12/61] arm64: kernel: Don't rely on objcopy to make code under pi/ __init From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071744_620304_360CC1E3 X-CRM114-Status: GOOD ( 13.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We will add some code under pi/ that contains global variables that should not end up in __initdata, as they will not be writable via the initial ID map. So only rely on objcopy for making the libfdt code __init, and use explicit annotations for the rest. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/pi/Makefile | 6 ++++-- arch/arm64/kernel/pi/kaslr_early.c | 16 +++++++++------- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index bc32a431fe35..2bbe866417d4 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -28,11 +28,13 @@ quiet_cmd_piobjcopy = $(quiet_cmd_objcopy) cmd_piobjcopy = $(cmd_objcopy) && $(obj)/relacheck $(@) $(<) $(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ \ - --remove-section=.note.gnu.property \ - --prefix-alloc-sections=.init + --remove-section=.note.gnu.property $(obj)/%.pi.o: $(obj)/%.o $(obj)/relacheck FORCE $(call if_changed,piobjcopy) +# ensure that all the lib- code ends up as __init code and data +$(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init + $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index b9e0bb4bc6a9..167081b30a15 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -17,7 +17,7 @@ #include /* taken from lib/string.c */ -static char *__strstr(const char *s1, const char *s2) +static char *__init __strstr(const char *s1, const char *s2) { size_t l1, l2; @@ -33,7 +33,7 @@ static char *__strstr(const char *s1, const char *s2) } return NULL; } -static bool cmdline_contains_nokaslr(const u8 *cmdline) +static bool __init cmdline_contains_nokaslr(const u8 *cmdline) { const u8 *str; @@ -41,7 +41,7 @@ static bool cmdline_contains_nokaslr(const u8 *cmdline) return str == cmdline || (str > cmdline && *(str - 1) == ' '); } -static bool is_kaslr_disabled_cmdline(void *fdt) +static bool __init is_kaslr_disabled_cmdline(void *fdt) { if (!IS_ENABLED(CONFIG_CMDLINE_FORCE)) { int node; @@ -67,17 +67,19 @@ static bool is_kaslr_disabled_cmdline(void *fdt) return cmdline_contains_nokaslr(CONFIG_CMDLINE); } -static u64 get_kaslr_seed(void *fdt) +static u64 __init get_kaslr_seed(void *fdt) { + static char const chosen_str[] __initconst = "chosen"; + static char const seed_str[] __initconst = "kaslr-seed"; int node, len; fdt64_t *prop; u64 ret; - node = fdt_path_offset(fdt, "/chosen"); + node = fdt_path_offset(fdt, chosen_str); if (node < 0) return 0; - prop = fdt_getprop_w(fdt, node, "kaslr-seed", &len); + prop = fdt_getprop_w(fdt, node, seed_str, &len); if (!prop || len != sizeof(u64)) return 0; @@ -86,7 +88,7 @@ static u64 get_kaslr_seed(void *fdt) return ret; } -asmlinkage u64 kaslr_early_init(void *fdt) +asmlinkage u64 __init kaslr_early_init(void *fdt) { u64 seed, range; From patchwork Tue Sep 12 14:16:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381768 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 79E51CA0EEB for ; Tue, 12 Sep 2023 14:18:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=V8FvdG48DcoDXedwvE+c8wKsjOiW6o5yF3fhmg+r45k=; b=mWxDECCXyt+93z2Lq340TJuWnX TrUjspNq++0rlw3wpNAzTRBf4EESMDZzjUE4jb2+OH39CxWnUTUApgAozmQIsfEnrQ8fpJoSA9A16 LK+8IC+3nq6JixWEjxXJYVBoCxqxlZPz2WUTxCoTvo9BxowouAu1a4e+ZGWKS8VsKZVRtbZmDBDZZ u5v4wCQc3rcpI/V0MSJiy8LpCXkmyFqUYbeEOWSYCFFZrMAKU+XxOuq2F3+nlRcUwYw/i/1LFysCC 813XGuXCqn6IQZkz1WvyuyFaIQael3R2ajSYd3q03tZSDSu60NV4imFi00dGXio/xCudegJ+mBYDj cJpbgAkA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DC-003WBf-1t; Tue, 12 Sep 2023 14:17:54 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4D5-003W3Y-0z for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:17:49 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-31fb093a53aso1182771f8f.0 for ; Tue, 12 Sep 2023 07:17:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528265; x=1695133065; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NY3EPoHR/3PQVRPFGLhWw3gqnjVe3x3PEJkh3SP9g28=; b=W/pTIqUPtkuBxzBX48IRSnLBWSxPKhYUdTOnwYMjkhqYXTF6VPOZvCQw2as7RFOx0t XQlj/BiXdBUdz0dTSUyZqlqyEOLHzwG3Zzr3bUHdPWzuByQwIfISiTcM+wm5jqvXooOt ydNuqdoWORZN8o0J5aYIhFZ4c2C1Vqht/mj8uVesCWr47c66tTqf7soguFKcP96xr4zr O0WIs6AiERkvCtXH9ORoMu533ynQMdYd9UlQCUML++wVhqMmATFtj8iRLeonfGZ0drR6 eUSdIUkBjTFxTWOpx9lcHVgDrb7Ymd76iKCDKh9ktgNNngmlODR2wMC7mRW6z2aRDYXM 1WWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528265; x=1695133065; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NY3EPoHR/3PQVRPFGLhWw3gqnjVe3x3PEJkh3SP9g28=; b=wN54HRdoGBZCKiIsTrL2F8cgD+rRWahXb+iBp1MdZCUSJakdA8hqh4JvZzYgcaQxpu qgb+V33kdOR5GqlGleiMlo8QQNVA8M2lNTulm5Wbxa/tx9rXUSEXyyX00gy/Jx5Z4vAF 5KqEgAMTJQRCU/BlAx5ZRsRhcvvYE+IqlNLaiQ0Seebe2YkdBXtBxgswD5F3iS7UkSow eS/BXyVb/G4injuL+IX7yfv9uwgKxfVwPhFlfIDK2Brv9HHg7SamKoGety1xhb5YB7N4 xV1SftUcaFQ+Zf2+lPneva5Y/uc/eQ/+xXD2T0OUOPsDnb6a/JOzDVP1z4x38mU/qn/N aMUg== X-Gm-Message-State: AOJu0YzZKGb7vicdZOUTRbweL/7/2fZCJYYIAcwGMgAFGjAH8plCRL7m DX9xTRWuyBmcz19ICP/3P9BsYgnj8fwpBdr1k5Ri76kpgRiU/hF4iMX6xDnUKfCZ/9QMbbMAb7g ssqIwgRl/69PK3N8JeGuxs5v3mDrVOa3fXurJXjscPACe4vpzEevBTn5ju57VBtVurd9ZY/tqbK w= X-Google-Smtp-Source: AGHT+IHMEaWwPuCOGyxkr4BJQ24vWr7py4B1Ffk422PmEo+bnk19Ltqi3AxGOB6y2Mn6EUXI4zNttHTJ X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:ec4c:0:b0:319:6fc6:151 with SMTP id w12-20020adfec4c000000b003196fc60151mr147729wrn.10.1694528265355; Tue, 12 Sep 2023 07:17:45 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:03 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10134; i=ardb@kernel.org; h=from:subject; bh=Meg/OwrqZQqGQSGLeqtq4LZV9QAh5/bMnwBF86I/jwA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaPHSNzGpCU0KR2y632xb9/Hr646/DgJvy06798yq2 SckNKu8o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEyE+z7DX8EVys5Jc7M/T9ta Hl+64YVxoOOeIFf/3L4tqWGxDwX3/mJk+Nt/0vftCtFD31JrHk9s4X/4wThjjb54s8euVS85m79 nsAAA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-76-ardb@google.com> Subject: [PATCH v4 13/61] arm64: head: move relocation handling to C code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071747_361352_C6B3CC07 X-CRM114-Status: GOOD ( 28.58 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that we have a mini C runtime before the kernel mapping is up, we can move the non-trivial relocation processing code out of head.S and reimplement it in C. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/Makefile | 3 +- arch/arm64/kernel/head.S | 104 ++------------------ arch/arm64/kernel/pi/Makefile | 5 +- arch/arm64/kernel/pi/relocate.c | 62 ++++++++++++ arch/arm64/kernel/vmlinux.lds.S | 12 ++- 5 files changed, 82 insertions(+), 104 deletions(-) diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index d95b3d6b471a..a8487749cabd 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -57,7 +57,8 @@ obj-$(CONFIG_ACPI) += acpi.o obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL) += acpi_parking_protocol.o obj-$(CONFIG_PARAVIRT) += paravirt.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o pi/ +obj-$(CONFIG_RELOCATABLE) += pi/ +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_ELF_CORE) += elfcore.o obj-$(CONFIG_KEXEC_CORE) += machine_kexec.o relocate_kernel.o \ diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index cab7f91949d8..a8fa64fc30d7 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -81,7 +81,7 @@ * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob - * x23 primary_entry() .. start_kernel() physical misalignment/KASLR offset + * x23 __primary_switch() physical misalignment/KASLR offset * x24 __primary_switch() linear map KASLR seed * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register @@ -389,7 +389,7 @@ SYM_FUNC_START_LOCAL(create_idmap) /* Remap the kernel page tables r/w in the ID map */ adrp x1, _text adrp x2, init_pg_dir - adrp x3, init_pg_end + adrp x3, _end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 mov_q x5, SWAPPER_RW_MMUFLAGS mov x6, #SWAPPER_BLOCK_SHIFT @@ -779,97 +779,6 @@ SYM_FUNC_START_LOCAL(__no_granule_support) b 1b SYM_FUNC_END(__no_granule_support) -#ifdef CONFIG_RELOCATABLE -SYM_FUNC_START_LOCAL(__relocate_kernel) - /* - * Iterate over each entry in the relocation table, and apply the - * relocations in place. - */ - adr_l x9, __rela_start - adr_l x10, __rela_end - mov_q x11, KIMAGE_VADDR // default virtual offset - add x11, x11, x23 // actual virtual offset - -0: cmp x9, x10 - b.hs 1f - ldp x12, x13, [x9], #24 - ldr x14, [x9, #-8] - cmp w13, #R_AARCH64_RELATIVE - b.ne 0b - add x14, x14, x23 // relocate - str x14, [x12, x23] - b 0b - -1: -#ifdef CONFIG_RELR - /* - * Apply RELR relocations. - * - * RELR is a compressed format for storing relative relocations. The - * encoded sequence of entries looks like: - * [ AAAAAAAA BBBBBBB1 BBBBBBB1 ... AAAAAAAA BBBBBB1 ... ] - * - * i.e. start with an address, followed by any number of bitmaps. The - * address entry encodes 1 relocation. The subsequent bitmap entries - * encode up to 63 relocations each, at subsequent offsets following - * the last address entry. - * - * The bitmap entries must have 1 in the least significant bit. The - * assumption here is that an address cannot have 1 in lsb. Odd - * addresses are not supported. Any odd addresses are stored in the RELA - * section, which is handled above. - * - * Excluding the least significant bit in the bitmap, each non-zero - * bit in the bitmap represents a relocation to be applied to - * a corresponding machine word that follows the base address - * word. The second least significant bit represents the machine - * word immediately following the initial address, and each bit - * that follows represents the next word, in linear order. As such, - * a single bitmap can encode up to 63 relocations in a 64-bit object. - * - * In this implementation we store the address of the next RELR table - * entry in x9, the address being relocated by the current address or - * bitmap entry in x13 and the address being relocated by the current - * bit in x14. - */ - adr_l x9, __relr_start - adr_l x10, __relr_end - -2: cmp x9, x10 - b.hs 7f - ldr x11, [x9], #8 - tbnz x11, #0, 3f // branch to handle bitmaps - add x13, x11, x23 - ldr x12, [x13] // relocate address entry - add x12, x12, x23 - str x12, [x13], #8 // adjust to start of bitmap - b 2b - -3: mov x14, x13 -4: lsr x11, x11, #1 - cbz x11, 6f - tbz x11, #0, 5f // skip bit if not set - ldr x12, [x14] // relocate bit - add x12, x12, x23 - str x12, [x14] - -5: add x14, x14, #8 // move to next bit's address - b 4b - -6: /* - * Move to the next bitmap's address. 8 is the word size, and 63 is the - * number of significant bits in a bitmap entry. - */ - add x13, x13, #(8 * 63) - b 2b - -7: -#endif - ret - -SYM_FUNC_END(__relocate_kernel) -#endif - SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir @@ -877,11 +786,11 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RELOCATABLE adrp x23, KERNEL_START and x23, x23, MIN_KIMG_ALIGN - 1 -#ifdef CONFIG_RANDOMIZE_BASE - mov x0, x22 - adrp x1, init_pg_end + adrp x1, early_init_stack mov sp, x1 mov x29, xzr +#ifdef CONFIG_RANDOMIZE_BASE + mov x0, x22 bl __pi_kaslr_early_init and x24, x0, #SZ_2M - 1 // capture memstart offset seed bic x0, x0, #SZ_2M - 1 @@ -894,7 +803,8 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, init_pg_dir load_ttbr1 x1, x1, x2 #ifdef CONFIG_RELOCATABLE - bl __relocate_kernel + mov x0, x23 + bl __pi_relocate_kernel #endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 2bbe866417d4..d084c1dcf416 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,5 +38,6 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o -extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) +obj-y := relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/pi/relocate.c b/arch/arm64/kernel/pi/relocate.c new file mode 100644 index 000000000000..1853408ea76b --- /dev/null +++ b/arch/arm64/kernel/pi/relocate.c @@ -0,0 +1,62 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Authors: Ard Biesheuvel +// Peter Collingbourne + +#include +#include +#include + +extern const Elf64_Rela rela_start[], rela_end[]; +extern const u64 relr_start[], relr_end[]; + +void __init relocate_kernel(u64 offset) +{ + u64 *place = NULL; + + for (const Elf64_Rela *rela = rela_start; rela < rela_end; rela++) { + if (ELF64_R_TYPE(rela->r_info) != R_AARCH64_RELATIVE) + continue; + *(u64 *)(rela->r_offset + offset) = rela->r_addend + offset; + } + + if (!IS_ENABLED(CONFIG_RELR) || !offset) + return; + + /* + * Apply RELR relocations. + * + * RELR is a compressed format for storing relative relocations. The + * encoded sequence of entries looks like: + * [ AAAAAAAA BBBBBBB1 BBBBBBB1 ... AAAAAAAA BBBBBB1 ... ] + * + * i.e. start with an address, followed by any number of bitmaps. The + * address entry encodes 1 relocation. The subsequent bitmap entries + * encode up to 63 relocations each, at subsequent offsets following + * the last address entry. + * + * The bitmap entries must have 1 in the least significant bit. The + * assumption here is that an address cannot have 1 in lsb. Odd + * addresses are not supported. Any odd addresses are stored in the + * RELA section, which is handled above. + * + * With the exception of the least significant bit, each bit in the + * bitmap corresponds with a machine word that follows the base address + * word, and the bit value indicates whether or not a relocation needs + * to be applied to it. The second least significant bit represents the + * machine word immediately following the initial address, and each bit + * that follows represents the next word, in linear order. As such, a + * single bitmap can encode up to 63 relocations in a 64-bit object. + */ + for (const u64 *relr = relr_start; relr < relr_end; relr++) { + if ((*relr & 1) == 0) { + place = (u64 *)(*relr + offset); + *place++ += offset; + } else { + for (u64 *p = place, r = *relr >> 1; r; p++, r >>= 1) + if (r & 1) + *p += offset; + place += 63; + } + } +} diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 3cd7e76cc562..8dd5dda66f7c 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -270,15 +270,15 @@ SECTIONS HYPERVISOR_RELOC_SECTION .rela.dyn : ALIGN(8) { - __rela_start = .; + __pi_rela_start = .; *(.rela .rela*) - __rela_end = .; + __pi_rela_end = .; } .relr.dyn : ALIGN(8) { - __relr_start = .; + __pi_relr_start = .; *(.relr.dyn) - __relr_end = .; + __pi_relr_end = .; } . = ALIGN(SEGMENT_ALIGN); @@ -317,6 +317,10 @@ SECTIONS init_pg_dir = .; . += INIT_DIR_SIZE; init_pg_end = .; +#ifdef CONFIG_RELOCATABLE + . += SZ_4K; /* stack for the early relocation code */ + early_init_stack = .; +#endif . = ALIGN(SEGMENT_ALIGN); __pecoff_data_size = ABSOLUTE(. - __initdata_begin); From patchwork Tue Sep 12 14:16:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381770 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 515E7CA0EED for ; Tue, 12 Sep 2023 14:18:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=VNEx2gd1sQjbxdNqL+LogZ2yOh3k0Pqg5criPg5cFVw=; b=MNxv9ftgFrr1gmRAGSnhG+zKqq m3sjBM3YYl19QkGrEARHRrLKDOn53yIrnv2bD7UFuZP0AP6v8beqPrlaAlBFvkZEukQH6EKb+QB0m f072xuEMeZbj/UGUaxf+bt5Vgk21mqLgDhfsNDXpRIORwreJqTgkv/4a2pmyq/E4QdWoDfAvXXNfh 9XTCqDDe+hVtOzSgJUS0w6D7SOkBspplG4KSg7Bj89DOMUzpQJZrPaou8sEUzFepDVCn9pv2LrlI/ bTF4caBsecwr67LB75UkxA+EsbvUAEtgjR/7iB1eTfgpB0DuUoD3MUJv2UKPtwIL+sC47S9H++DMq bZ4nWJQA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DF-003WEO-23; Tue, 12 Sep 2023 14:17:57 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4D7-003W69-2C for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:17:51 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-31aed15ce6fso3513848f8f.3 for ; Tue, 12 Sep 2023 07:17:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528268; x=1695133068; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mj2P9EYaaemecU+L4Ww9YzwcvsDLGOKJqjgIcK+uvQU=; b=IKex2gy0ftuZJC6a9zdyGiITNY7IvqSyITX5/YTqnhFR5pPwovZTSSojpykwxu+vni oLToWGnsK9OHGwyDiPWwncSSeOec83lfcMXgouUtyVaCUo00crVCJhOa+q5JVPlL+7CW 4hfSs7jsEuSEweDZIk2v91kcJmPEeRisnD31l9rAg2HfNZkk3pVlL7CjwTM3hjg74SoV kHsvH3D3H0tXc3Llju2rAw28BVR3uvtkhC8ijZAESM2s18lUoaD5rcaS6pMx6pjNOSPh 3h+o2a1Ly1FiA4emKf8Oy5nXvv0ePT/4wasRNK/7Tq5kqd9Li37juEswzAfT8zxzRfPl x26A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528268; x=1695133068; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mj2P9EYaaemecU+L4Ww9YzwcvsDLGOKJqjgIcK+uvQU=; b=j6uOoajRLoejvmDpXWXqIK4q2yZKdAQQ6eNSqkKEQolZYGqJD5JHRGKir+27iu/lP5 fir59jUOUIyy+lxJYVpvasHPVSzx0Uk/rBqykt6HXJ64FUu+PjmYfCQtj2g+T7mwEH8n XG6mqAEE/OV4K9An6gDRaEpar5MmWTVOakqdv4+5dE9N1je52demhnuP/DTFm5hlKFJF FvCvT+I6BWoiyhXqqih5RcvLZiTp/3SG/hX+fYbVjiHokdHJIeMB8dVHaevRoMniPrVh clfiII93mzWzrVaNZOvZM7MzwX1YgXHsTcjhA953mM5RHWJcmBrVGWBZBwKcg6IWwwEb 2h2w== X-Gm-Message-State: AOJu0YwIS0sPrltKg2Fcpv1LyiCmpsdygMVsxcTGDXbEpd9pY+4oEtuu Z1HaMT8iXKy2XjLP+UgomkOQs9iNYqD64BZkGkn8/sb+kHUFSagaeHSKM3SbrRrOBi6hx66vkTj lRHb/FVcuvtCh/Y/pBGRE4aikvTgN179YBpX/4dvzF6hVOVMsAGZ7rITP2h08Tp0LyNi1sYTUwY 4= X-Google-Smtp-Source: AGHT+IFO3sKfQu7yWaktt4mXC+tWM2QANfyhmY8U24KJc8wtWKcTjvYf/hWg6g3/SCgfIgdq8tGpNWKM X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:68ca:0:b0:317:3c15:b589 with SMTP id p10-20020a5d68ca000000b003173c15b589mr141873wrw.11.1694528267650; Tue, 12 Sep 2023 07:17:47 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:04 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1685; i=ardb@kernel.org; h=from:subject; bh=5r9ZDvRf/5Qx3wG13F8lcLnVag+F7odStGTabgSgb8U=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaAl7ieeuGuuD/Z5vcq9pSfy/xXRmuyxXu8lm/8fnZ 5crND/qKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABM5WMLwP2h5ysWK2UWe1/mv pHYXtIdLHGnaWlgVWlS0tlm7SYk7npFh4bX52ntkHjud/86cdazGssX1+r0tp5raS5ovLnjdcNC PBwA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-77-ardb@google.com> Subject: [PATCH v4 14/61] arm64: idreg-override: Omit non-NULL checks for override pointer From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071749_723897_9A6337D8 X-CRM114-Status: GOOD ( 14.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that override pointers are always set, we can drop the various non-NULL checks that we have in the code. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index 3addc09f8746..536bc33859bc 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -216,9 +216,6 @@ static void __init match_options(const char *cmdline) for (i = 0; i < ARRAY_SIZE(regs); i++) { int f; - if (!regs[i]->override) - continue; - for (f = 0; strlen(regs[i]->fields[f].name); f++) { u64 shift = regs[i]->fields[f].shift; u64 width = regs[i]->fields[f].width ?: 4; @@ -319,10 +316,8 @@ asmlinkage void __init init_feature_override(u64 boot_status) int i; for (i = 0; i < ARRAY_SIZE(regs); i++) { - if (regs[i]->override) { - regs[i]->override->val = 0; - regs[i]->override->mask = 0; - } + regs[i]->override->val = 0; + regs[i]->override->mask = 0; } __boot_status = boot_status; @@ -330,9 +325,8 @@ asmlinkage void __init init_feature_override(u64 boot_status) parse_cmdline(); for (i = 0; i < ARRAY_SIZE(regs); i++) { - if (regs[i]->override) - dcache_clean_inval_poc((unsigned long)regs[i]->override, - (unsigned long)regs[i]->override + - sizeof(*regs[i]->override)); + dcache_clean_inval_poc((unsigned long)regs[i]->override, + (unsigned long)regs[i]->override + + sizeof(*regs[i]->override)); } } From patchwork Tue Sep 12 14:16:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381771 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 231ACCA0EEB for ; Tue, 12 Sep 2023 14:18:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=+rOBwHODUr9HlHhX1cfjEOkBqEenZgZkZVwdUmbVbw8=; b=TYEgfLQxqxwtiShMSnMGeoTfrl x10oePzWVpONmTbt9XK6/fa8JENrGSqphza5zacEwqIdFZsqwKNhr7FvuBXCMWSyDrIBuFoh4cfvl iC9VuhaCoeCnC9lHb9Ucc4to75Q9Ephn4aXeYnX6z+pYaaWCpSuCrPp3jCi39249IAIUNMEgiwpQ2 1z+JaTKIOFNhT2z62GVHEFE7HaX5nIAm5QgdwafvGYgh/FAFZ56dEWuA8IAaDgCB9NM00uCnkX4V+ 5vbErlfV/YJzAUjRCQCP6cP8IN2eORI+alNs1/NHHNja+lnwCRyCAmsUNAmCxa+A7iRGYIs9nh134 xhxl+d2w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DM-003WKV-0l; Tue, 12 Sep 2023 14:18:04 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DA-003W8V-0Z for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:17:53 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-31ad77537ebso3755983f8f.0 for ; Tue, 12 Sep 2023 07:17:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528270; x=1695133070; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=02lVXddcaK3RQ83APdc0qFSnsO4qXZyNWprlqvJrxN8=; b=NbG/3Jrp80KYd7/wrc366/rQvZ/DFhgjpE4k8DJRs4Kx25ZIzlo6wn3DNSQR+jLsb3 aUsvq67eZAojzpv+Rgh1OSYTyivUkYI3vgJ/MZuH6kaDh73+E2uUx2iNt/nc6Tjfw7ro zyYfb29kjUJcjhTcpH7ycHYoJt7ST7eJyJSxSLD2r6fidGEdNe0un91ehlRFSlVMLbSa 9XFPsNGuN73SZCOmvyDy9HQXOB9QJ8V0TCA7Rb7oRYVH25a9GK9O72GWdtqZxKcmBiK0 R56WXXstgYAZgrdvWpXGj6GiLaX34mDyRK7z1IXWZkqh7snRiwPwQ0dIlZo4Aq1tDIuv LeRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528270; x=1695133070; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=02lVXddcaK3RQ83APdc0qFSnsO4qXZyNWprlqvJrxN8=; b=GbOiIa+x/SuZUjNWmwiVu2CzRyQLwQUJofGTeXN1HImHW5nIGmZmiW9iF3qP6Svw2t GNggFckkVR2FAI8u3cH3qbXbneI6Ghmy4YJcQEqpf96syjTdHe3kN9cLSDTa+rYXWRz+ f0aJ1HI/ON+yXM0dCYy6aXOeeWE9B3UzSJO+egxwqYulXumOaAZJ+yW6DxiYlhfp5AcA 4Wxr4PhmnQBPWby7Wd2AnoeZH9ewdbZ0ViaavSUamR4uDJl37YXZMu3475T3HpfhoEmj 2dWn9Z7Dk9XUHJyHtKmXgVWDbpexeYiKEtE18WvWJW9/1CtDlOEWXhgt/csDUDWkKyX9 nHrA== X-Gm-Message-State: AOJu0YxJh8N+8pY2XJmDTZvfWMw/RXsAqikn/5zTFKDE4wM89R/7t1WQ L0VkkDothRz0XSyMzUowytPYzMqVrusn49JXhexnqordCt8r0d7YBJx0s5QSMOCMRr76H5Q6YBS nvTP8F0XGWFh0vdjmeApedycKKj5AjWk33YtviAc9hJmeSx1Kuc+EVqYf/y+QeuBUpNV89goYI0 g= X-Google-Smtp-Source: AGHT+IE7V3PNzWt5wMTJ4+as401aV2F2tm1W5Qj9hLMpmVv0/Egh+h43dSdUsq8xpH9cmmKVikx9UQas X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:61c6:0:b0:318:bfe:455b with SMTP id q6-20020a5d61c6000000b003180bfe455bmr146916wrv.5.1694528270075; Tue, 12 Sep 2023 07:17:50 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:05 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=7404; i=ardb@kernel.org; h=from:subject; bh=VHwbQklrHHarXJdq4lwq90XZWzEI7OYiEMQKAxOXrSk=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaKn3vmC3hTeycrR2LVxQ/GFlpcraM28SWeKsnab+2 aWuX5DeUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACaieoeRoTVt6tYpS89Y3XAt rDc+uapkjp/K16lqm3l25BXKeh84LM7wv/z0Co6UFOH0F4evrZuz7krdo03rp64NWfguel3DrgV XpjIDAA== X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-78-ardb@google.com> Subject: [PATCH v4 15/61] arm64: idreg-override: Prepare for place relative reloc patching From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071752_214189_DFC3771B X-CRM114-Status: GOOD ( 22.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The ID reg override handling code uses a rather elaborate data structure that relies on statically initialized absolute address values in pointer fields. This means that this code cannot run until relocation fixups have been applied, and this is unfortunate, because it means we cannot discover overrides for KASLR or LVA/LPA without creating the kernel mapping and performing the relocations first. This can be solved by switching to place-relative relocations, which can be applied by the linker at build time. This means some additional arithmetic is required when dereferencing these pointers, as we can no longer dereference the pointer members directly. So let's implement this for idreg-override.c in a preliminary way, i.e., convert all the references in code to use a special accessor that produces the correct absolute value at runtime. To preserve the strong type checking for the static initializers, use union types for representing the hybrid quantities. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 98 +++++++++++++------- 1 file changed, 65 insertions(+), 33 deletions(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index 536bc33859bc..4e32a44560bf 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -21,14 +21,32 @@ static u64 __boot_status __initdata; +// temporary __prel64 related definitions +// to be removed when this code is moved under pi/ + +#define __prel64_initconst __initconst + +typedef void *prel64_t; + +static void *prel64_to_pointer(const prel64_t *p) +{ + return *p; +} + struct ftr_set_desc { char name[FTR_DESC_NAME_LEN]; - struct arm64_ftr_override *override; + union { + struct arm64_ftr_override *override; + prel64_t override_prel; + }; struct { char name[FTR_DESC_FIELD_LEN]; u8 shift; u8 width; - bool (*filter)(u64 val); + union { + bool (*filter)(u64 val); + prel64_t filter_prel; + }; } fields[]; }; @@ -46,7 +64,7 @@ static bool __init mmfr1_vh_filter(u64 val) val == 0); } -static const struct ftr_set_desc mmfr1 __initconst = { +static const struct ftr_set_desc mmfr1 __prel64_initconst = { .name = "id_aa64mmfr1", .override = &id_aa64mmfr1_override, .fields = { @@ -70,7 +88,7 @@ static bool __init pfr0_sve_filter(u64 val) return true; } -static const struct ftr_set_desc pfr0 __initconst = { +static const struct ftr_set_desc pfr0 __prel64_initconst = { .name = "id_aa64pfr0", .override = &id_aa64pfr0_override, .fields = { @@ -94,7 +112,7 @@ static bool __init pfr1_sme_filter(u64 val) return true; } -static const struct ftr_set_desc pfr1 __initconst = { +static const struct ftr_set_desc pfr1 __prel64_initconst = { .name = "id_aa64pfr1", .override = &id_aa64pfr1_override, .fields = { @@ -105,7 +123,7 @@ static const struct ftr_set_desc pfr1 __initconst = { }, }; -static const struct ftr_set_desc isar1 __initconst = { +static const struct ftr_set_desc isar1 __prel64_initconst = { .name = "id_aa64isar1", .override = &id_aa64isar1_override, .fields = { @@ -117,7 +135,7 @@ static const struct ftr_set_desc isar1 __initconst = { }, }; -static const struct ftr_set_desc isar2 __initconst = { +static const struct ftr_set_desc isar2 __prel64_initconst = { .name = "id_aa64isar2", .override = &id_aa64isar2_override, .fields = { @@ -128,7 +146,7 @@ static const struct ftr_set_desc isar2 __initconst = { }, }; -static const struct ftr_set_desc smfr0 __initconst = { +static const struct ftr_set_desc smfr0 __prel64_initconst = { .name = "id_aa64smfr0", .override = &id_aa64smfr0_override, .fields = { @@ -149,7 +167,7 @@ static bool __init hvhe_filter(u64 val) ID_AA64MMFR1_EL1_VH_SHIFT)); } -static const struct ftr_set_desc sw_features __initconst = { +static const struct ftr_set_desc sw_features __prel64_initconst = { .name = "arm64_sw", .override = &arm64_sw_feature_override, .fields = { @@ -159,14 +177,17 @@ static const struct ftr_set_desc sw_features __initconst = { }, }; -static const struct ftr_set_desc * const regs[] __initconst = { - &mmfr1, - &pfr0, - &pfr1, - &isar1, - &isar2, - &smfr0, - &sw_features, +static const union { + const struct ftr_set_desc *reg; + prel64_t reg_prel; +} regs[] __prel64_initconst = { + { .reg = &mmfr1 }, + { .reg = &pfr0 }, + { .reg = &pfr1 }, + { .reg = &isar1 }, + { .reg = &isar2 }, + { .reg = &smfr0 }, + { .reg = &sw_features }, }; static const struct { @@ -214,15 +235,20 @@ static void __init match_options(const char *cmdline) int i; for (i = 0; i < ARRAY_SIZE(regs); i++) { + const struct ftr_set_desc *reg = prel64_to_pointer(®s[i].reg_prel); + struct arm64_ftr_override *override; int f; - for (f = 0; strlen(regs[i]->fields[f].name); f++) { - u64 shift = regs[i]->fields[f].shift; - u64 width = regs[i]->fields[f].width ?: 4; + override = prel64_to_pointer(®->override_prel); + + for (f = 0; strlen(reg->fields[f].name); f++) { + u64 shift = reg->fields[f].shift; + u64 width = reg->fields[f].width ?: 4; u64 mask = GENMASK_ULL(shift + width - 1, shift); + bool (*filter)(u64 val); u64 v; - if (find_field(cmdline, regs[i], f, &v)) + if (find_field(cmdline, reg, f, &v)) continue; /* @@ -230,16 +256,16 @@ static void __init match_options(const char *cmdline) * it by setting the value to the all-ones while * clearing the mask... Yes, this is fragile. */ - if (regs[i]->fields[f].filter && - !regs[i]->fields[f].filter(v)) { - regs[i]->override->val |= mask; - regs[i]->override->mask &= ~mask; + filter = prel64_to_pointer(®->fields[f].filter_prel); + if (filter && !filter(v)) { + override->val |= mask; + override->mask &= ~mask; continue; } - regs[i]->override->val &= ~mask; - regs[i]->override->val |= (v << shift) & mask; - regs[i]->override->mask |= mask; + override->val &= ~mask; + override->val |= (v << shift) & mask; + override->mask |= mask; return; } @@ -313,11 +339,16 @@ void init_feature_override(u64 boot_status); asmlinkage void __init init_feature_override(u64 boot_status) { + struct arm64_ftr_override *override; + const struct ftr_set_desc *reg; int i; for (i = 0; i < ARRAY_SIZE(regs); i++) { - regs[i]->override->val = 0; - regs[i]->override->mask = 0; + reg = prel64_to_pointer(®s[i].reg_prel); + override = prel64_to_pointer(®->override_prel); + + override->val = 0; + override->mask = 0; } __boot_status = boot_status; @@ -325,8 +356,9 @@ asmlinkage void __init init_feature_override(u64 boot_status) parse_cmdline(); for (i = 0; i < ARRAY_SIZE(regs); i++) { - dcache_clean_inval_poc((unsigned long)regs[i]->override, - (unsigned long)regs[i]->override + - sizeof(*regs[i]->override)); + reg = prel64_to_pointer(®s[i].reg_prel); + override = prel64_to_pointer(®->override_prel); + dcache_clean_inval_poc((unsigned long)override, + (unsigned long)(override + 1)); } } From patchwork Tue Sep 12 14:16:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381772 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 837D7CA0EEC for ; Tue, 12 Sep 2023 14:18:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=m+ub3uoiLaxqybovPaZ7xQfc/0XZmovBLA9csxgjvlM=; b=t2DmjvIrI2KK2bjzz1dmAXTvHq rX8Ek/PEBd6X6xM0KSaEUwBU3q7F7O8WW0NaJxH6rS9jISCglW2hc/FO8378zSZmaiJrfPcyPTfSa 2vaDmJjJvt4PXXUEwrWjzyOPZZC6MzT/A2gYAyyJESgxLRYWY7d9RX6TM1TdGFOnXA96dsqtE7GHt snaj51Yv0xeiLjqLXSZBpAzIMxn2HbHWGE7AM4vPH4m8xBSyPwf5m00wbKbqDucsbXJUThieUReHg Je3ErChPrmOs4f6/2iQR3XLoleUdC5q//aHRXYzTCLgFA7CqDrmqkV60G/B05DesgzH7TEGZ08NH8 XTmPckIw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DR-003WP6-0E; Tue, 12 Sep 2023 14:18:09 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DC-003WAb-1w for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:17:56 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-31f3eaa5c5eso3751603f8f.3 for ; Tue, 12 Sep 2023 07:17:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528272; x=1695133072; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hHYwcfo64DWIAT1yMYOXBOJJh5fY950vXfntgtoofFU=; b=IIGJ0/11/PpH/kqwhX091CWIC1PVRvCimAKrwScE+e7LIDwbrlYoFFitmM+Hr+hffr GfJrfLDDq+eXL84DJl7zbECusFjX3goARZ6M+4KJpniku+IYCTE3ADjiniRyvEJPVwYP VGFhq4K2MS1LkBTkJa/Vo0EAdbZ9+gvJkVSgXbByIULD2KpYp41+FelZp+z90GeEENu3 ZHbDY8ETTz0YUDt0LzaCxBz7pVsn/rFSTsDw/hOsg6shOjCxlsozVgMule5ri+9PP6f6 ZuhkVSIZHqsSrobdlekG4ZM46/2iU5q2f5KsqcVlFf7OX1axOcfoyEJ5fkq+TfurrhCD p5/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528272; x=1695133072; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hHYwcfo64DWIAT1yMYOXBOJJh5fY950vXfntgtoofFU=; b=ip4ws40jTtZ4rYa3IuBeBgt2CO0fno1WH0PmqRiOAnU9pVS4COjM/C/gGr6vaL80hO gnw0l/ItXE+yh2EW4GanaxYYY/87MJJDLi6c3HCubwPwtndtRlkiijK/pv7kRiMg7fm5 4iaNPgrv6RFOjbT3tRQLN3X/RsOci4d/0xssiZfGjv/+ufjt//uijrE2zjBAOgHMB/Zz RhyJ0RO44gOm7ukjzpfiAgg0qGyyLwBkNdcCe1ve1Vqgi7BxoQxvxCePTT6OEEKHRsvt rwPy4AUpbVCDQWHSk+Yg5HXfNQnWiV8/G3nsrNcKeNu8IURYsjfmkQuUr4cqIWCavHkF Z+8A== X-Gm-Message-State: AOJu0Ywg3QHs94dSS2dzpGEBYSgthq3l9FkhYiLYLD9cDV+XW4y0LjfU Liu7St/spVwdybNbYYp3gfX4wH2rKWltarUGrUYUm7Td0Vfdu2sBXptnZfVAdYzZDfUL9ts+zsr qbHknDyWTuv34PO65Zg0q1rVm2e2fuN/uqhIuJ/cgfBvU13H/O3w1TfYUXUhY3kSAExUcqIh3x6 I= X-Google-Smtp-Source: AGHT+IELYFm2RlyeJmSe5h8QziYum/ySAT8PDWVFq9ML6m/Kbt/NSP0/wVURPIEroUf9iFjdy1lG+tnc X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:660c:0:b0:317:5181:90ab with SMTP id n12-20020a5d660c000000b00317518190abmr152393wru.3.1694528272330; Tue, 12 Sep 2023 07:17:52 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:06 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2763; i=ardb@kernel.org; h=from:subject; bh=CgRFQOMG2Yq3syft2Z3nHkoox+JzGIJN6TqHLAQ6JNc=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaNnuwo9iXEf2JQpk3dy1dd/VyKhgzrMVD30MD+2uv 8sVNGltRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZiISgbDP3WOjAms3Sw7vs/i ufl06ZPZG6oVtfZcjzm+7y975wquozwM/zM3SKc+9eL7G6EWn80R6Zp01eHO99a6lKv1NtcX1ci yMAAA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-79-ardb@google.com> Subject: [PATCH v4 16/61] arm64: idreg-override: Avoid parameq() and parameqn() From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071754_685136_20026AC4 X-CRM114-Status: GOOD ( 19.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The only way parameq() and parameqn() deviate from the ordinary string and memory routines is that they ignore the difference between dashes and underscores. Since we copy each command line argument into a buffer before passing it to parameq() and parameqn() numerous times, let's just convert all dashes to underscores just once, and update the alias array accordingly. This also helps reduce the dependency on kernel APIs that are no longer available once we move this code into the early mini C runtime. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 28 ++++++++++++-------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index 4e32a44560bf..cb4de31071d4 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -194,8 +194,8 @@ static const struct { char alias[FTR_ALIAS_NAME_LEN]; char feature[FTR_ALIAS_OPTION_LEN]; } aliases[] __initconst = { - { "kvm-arm.mode=nvhe", "id_aa64mmfr1.vh=0" }, - { "kvm-arm.mode=protected", "id_aa64mmfr1.vh=0" }, + { "kvm_arm.mode=nvhe", "id_aa64mmfr1.vh=0" }, + { "kvm_arm.mode=protected", "id_aa64mmfr1.vh=0" }, { "arm64.nosve", "id_aa64pfr0.sve=0" }, { "arm64.nosme", "id_aa64pfr1.sme=0" }, { "arm64.nobti", "id_aa64pfr1.bt=0" }, @@ -224,7 +224,7 @@ static int __init find_field(const char *cmdline, len = snprintf(opt, ARRAY_SIZE(opt), "%s.%s=", reg->name, reg->fields[f].name); - if (!parameqn(cmdline, opt, len)) + if (memcmp(cmdline, opt, len)) return -1; return kstrtou64(cmdline + len, 0, v); @@ -281,23 +281,29 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) cmdline = skip_spaces(cmdline); - for (len = 0; cmdline[len] && !isspace(cmdline[len]); len++); - if (!len) + /* terminate on "--" appearing on the command line by itself */ + if (cmdline[0] == '-' && cmdline[1] == '-' && isspace(cmdline[2])) return; - len = min(len, ARRAY_SIZE(buf) - 1); - memcpy(buf, cmdline, len); - buf[len] = '\0'; - - if (strcmp(buf, "--") == 0) + for (len = 0; cmdline[len] && !isspace(cmdline[len]); len++) { + if (len >= sizeof(buf) - 1) + break; + if (cmdline[len] == '-') + buf[len] = '_'; + else + buf[len] = cmdline[len]; + } + if (!len) return; + buf[len] = 0; + cmdline += len; match_options(buf); for (i = 0; parse_aliases && i < ARRAY_SIZE(aliases); i++) - if (parameq(buf, aliases[i].alias)) + if (!memcmp(buf, aliases[i].alias, len + 1)) __parse_cmdline(aliases[i].feature, false); } while (1); } From patchwork Tue Sep 12 14:16:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381773 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 95682CA0EEB for ; Tue, 12 Sep 2023 14:18:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=/55ANkpFs9/L8ePj6FiWfIZfwXEcl6bt95wccZXxgs0=; b=ENXz0c46WOGvlIDB3DIPYtmq2F 1ixa/Yqf7mC6GfahlnwBi4IfNM7Zs3JRuFcf6ni4QPjntyKd6Jr9TcK63vHKjuO0D0If28ru0z4jo kCGLq1r6I3Ce+ExCcZvnLDDkbg1bYcAVWaMYuVdGxTKOBX+hJ6geH5T61CzYAsthipiiBSQzBPSo0 SEUM26gEz9MoObQ9u/cWiTxZewekn2rSxEED/AN+m0CyEqKAGZyEP1B7FLy9NUR+Q7O4fiv1v9xMY mU8QyDOVz5qtdrnQci7amK0K6yAnNa1bd7TPoQLiniI8yoKQvoO64iRpflxg00B8jmuGEt3PWJaqk YQneoXBg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DV-003WSn-33; Tue, 12 Sep 2023 14:18:13 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DF-003WD4-1r for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:01 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-31f3cfe7269so3743838f8f.2 for ; Tue, 12 Sep 2023 07:17:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528275; x=1695133075; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/sepClc9hwFiPgenvTU2qpDKp7M1L/k4QwNRfj3ODTg=; b=GqK6JM5BlMfvcwkXW1WYUVrJnx9iNQjLSZeKLZ9QyzGzH2xaEog0ZBad+7Jy8LnfhJ mAtNt2fnVmERYFKAKBiOezbih/fIg0rK1EaBsZ9sCFTp0WCfJazafp/wA/Li6+CMi2aO 2ZHtVktcHDbzfCs040Te4tXtfMXzg0upC+88mKd3qC7D17p7+lEOQprrZ5HG/SZtuKC4 oLKMENnz6jgibarh9goNChEiDPLO2EErE/prRKKExk205V+E6PSVxBxSLLo0Z7rLr1tG mH1CL3Obu3gGKYM+s7cBcpkdkL8kXIPqLQD6gRKyQjKj8g/cGCemNsHlIjB9AsLBSM6r Xn9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528275; x=1695133075; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/sepClc9hwFiPgenvTU2qpDKp7M1L/k4QwNRfj3ODTg=; b=PC83N0DAAC4MTlLt3GnaJloX1bDhhrGokVkpiCsKAKwWD1wITytX0C+RZc/SVs8w8v BJvsogrhf/tOHJcUzh77T4c2FE9RyHRsZD2lJ6H+K/7T0YILO7DwyIB1FEfrnx6VD043 snQ7rjUeWJ2PJPh3f4viNTSgtHvqWiv6zKuvqYuMA/y+6qsDGOFqDQOWzpeL7kKtgFwJ p7bfap8ejWXwHTng8OOHpK5CDe4CGCUEH1HWD8D+gtCYOZDpdFuXmEJOIBPh2oAqdpX5 2zulo1W34kRZ7M0zsU8m0T+gAA1a06Syeb3CliCvYFLyfCREIMzlLUsYeDvji7d8S8zB SUkQ== X-Gm-Message-State: AOJu0YyB/XTBlVWPT+ZFOn9MaHfvXHo+9tlH/qYysUdq+An08UfZJSOv 2XsYv98tiwiNX0WW0uXTntCHrx553y3Zt2G4nEFRH0YhlrkheCeFXeFYkmDGdhgSNSLI4wnp0DS nToVZHp0FJHW4HOsDKal/0K89RELxUziDG5lIwC6EsHIyTQSteIQnJEhLbyeUY2l3FrEvQ6U/PT g= X-Google-Smtp-Source: AGHT+IHU8gMmS5TwfOqAZEV4MjyvKtzaur0Ea0HW6oC2uwUGkNkOkG8swbTeO/JQUaCRTxmKI+lC/uNB X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:14b:b0:313:f2bf:4c43 with SMTP id r11-20020a056000014b00b00313f2bf4c43mr143906wrx.2.1694528275050; Tue, 12 Sep 2023 07:17:55 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:07 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=924; i=ardb@kernel.org; h=from:subject; bh=7dwN4994wU2RkOX6xb0W36D2zJFK5xDHZIUcogKBXeY=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaHm48O8Maf/Vf5q17pVe+yT/kD+rbX5QejqPYWVcn bJf3uWOUhYGMQ4GWTFFFoHZf9/tPD1RqtZ5lizMHFYmkCEMXJwCMJGYXkaGY8q5T3g2fKtfYlbo Zz+f0TS42qHlWs0Gm6fZ/okLj+nHMPxTerTw7VeVLIPp670Uf7zqYso6p/UxZ72XZsDJQNZUES8 mAA== X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-80-ardb@google.com> Subject: [PATCH v4 17/61] arm64: idreg-override: avoid strlen() to check for empty strings From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071757_669030_93D2B550 X-CRM114-Status: GOOD ( 14.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel strlen() is a costly way to decide whether a string is empty, as in that case, the first character will be NUL so we can check for that directly. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index cb4de31071d4..96fec5b7a65e 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -241,7 +241,7 @@ static void __init match_options(const char *cmdline) override = prel64_to_pointer(®->override_prel); - for (f = 0; strlen(reg->fields[f].name); f++) { + for (f = 0; reg->fields[f].name[0] != '\0'; f++) { u64 shift = reg->fields[f].shift; u64 width = reg->fields[f].width ?: 4; u64 mask = GENMASK_ULL(shift + width - 1, shift); From patchwork Tue Sep 12 14:16:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381863 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 18633CA0EF2 for ; Tue, 12 Sep 2023 15:30:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=xs6o+T3XhtODyqLxVZXVq8T/BFe0/95q061oYDWVna0=; b=jcxMmaXpbV4QrbS2wpH6md3afX 5d7v3YAqFWLZvHSi6mJJprht/GNA0ewdzvMC7Zn+oiNhBvpzjRdym+hZbDHtBdPXG8VbjCQHUcTdG 3G/RkFTwxuA3RvCg+FpfuwvDdC8uNWsfp25Htk+nX5HAwuXYfU28ObSR1prlCaFSBUTcYGAch6otC RDyZ0kFZqsl0IpBaiYlI1oE6Y+jBtKSGSR9YKtLiV5B24bZGa9D5LXN37K6d2KfcQ5w/11RHFmBbK n03IFkLiFaHgy/gJ6/1DWSKGpkZVg9QaeCnVj/EYKJkEfbyqrENCQVzHqQJOiMtlbwP90ODkXuOse iYh6m0PA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg5LP-003igF-0r; Tue, 12 Sep 2023 15:30:27 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DH-003WFS-33 for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:07 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-4011fa32e99so43408345e9.0 for ; Tue, 12 Sep 2023 07:17:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528277; x=1695133077; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Pvdt5Oe1T19y8sApBOZg/loa9q7iUxAgj+0AqF1CeGg=; b=b8nJvdgUNR+y6dljIaO7xAC2HgdnmVjgi2LfMcbAxxD1be5Q2TkHoyaR/ZodZKq6SE bbEpnAUKe/NE1mElMV4y8lQnpKfljyNFG1TqVZ6+xo3mLl6aadxHZLvpQHsACiAfFk4a UeyrCAl1Vlg6KybUwB7hReyyeztZhfDpJqJD8IBz6o9KScQ6RHO6PSfr3J38fDlghSFB 8KloqKWLKzrVetJsu+3LsTBUumZ7oHJqOSNv3Ji04AOD2xx5g3f/lT4xnLQ6BL3MSiLL 19H5DrLMM2CQMl2WouChmD08W7RRijwRvq7cGI2mp8nty6QNV/f15KgP4JffuK85/ndG 74tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528277; x=1695133077; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Pvdt5Oe1T19y8sApBOZg/loa9q7iUxAgj+0AqF1CeGg=; b=v4c7CN3nEXDXfUcuoei/VsJiHJSud26lY2yBA5F+zTh3YZ6rWqDpBKAGopX3cZqkM+ UapKMf6kih47g5ojBUdSAOcs0QwWAtPIdB7uryBPifTffaSsjL1zl9ceP1J8bwP01Xgn EqtMmYOk+WjRh8sZEh2owp2cONroCp7r+BbcgI8lYbjit3ZcMFKwEKjQnotZOJxIzKCS BkWfFkHcy95qrcT3eMf+TbzJBuNOH7OlMIrNsflcnSS4bv6owqJ225TO8KjEUEfxHLqs Bd2xKE96bCiKigA50PQrhYNB2esjLYJsGNCd4r/z4HQLnNxnQCM8Y3szFt89kgYFgPSV By+w== X-Gm-Message-State: AOJu0Yymxj+nhdRhQDy5ONxOxIguxmy52GXDG57LmjbFcABgKmU2Y8+V /3Gy9uXdqC+pKMpbEwq6L5jsvNgkJXZIPFZ6fz9NeI+/8Dh5XuwttQAvo+Fot8lulIfeUVwIp8M llsSlAiosSUuB7ugLF/KxJWjJ7Il/mdBMvU8gUyPXEw5RjOGny62tsm2sYs0AmbrmglHKyKAriw 8= X-Google-Smtp-Source: AGHT+IEDnRLHFP0vznFo5jDa0GsBziy1byoBu7OS7lDJJzGkX6MAuqjU7f2zKsSIysFC/7L7Pqhx+LnH X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:4388:0:b0:317:6b94:b700 with SMTP id i8-20020a5d4388000000b003176b94b700mr147658wrq.9.1694528277588; Tue, 12 Sep 2023 07:17:57 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:08 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2430; i=ardb@kernel.org; h=from:subject; bh=t58xkJ8N9gZCd7n1STbdJu5HNYTUXfHvSs7cQZ8K738=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaIVNRdR8z2kVf+SPCUY7Tc3c9cn+udFeLa0bLbcth Q6selbaUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACayVpaRoUsplc3kIMsLtc+M Zi4mRrsfn/zh8IXJ+k6Cp0Tho+q3JowMO758XP1/ckmM4RqLT4s+vu3y2xyXUqfd3LaKozPoDq8 +IwA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-81-ardb@google.com> Subject: [PATCH v4 18/61] arm64: idreg-override: Avoid sprintf() for simple string concatenation From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071800_014531_723CBAAB X-CRM114-Status: GOOD ( 17.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Instead of using sprintf() with the "%s.%s=" format, where the first string argument is always the same in the inner loop of match_options(), use simple memcpy() for string concatenation, and move the first copy to the outer loop. This removes the dependency on sprintf(), which will be difficult to fulfil when we move this code into the early mini C runtime. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index 96fec5b7a65e..41eeca857b32 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -215,14 +215,15 @@ static int __init parse_nokaslr(char *unused) } early_param("nokaslr", parse_nokaslr); -static int __init find_field(const char *cmdline, +static int __init find_field(const char *cmdline, char *opt, int len, const struct ftr_set_desc *reg, int f, u64 *v) { - char opt[FTR_DESC_NAME_LEN + FTR_DESC_FIELD_LEN + 2]; - int len; + int flen = strlen(reg->fields[f].name); - len = snprintf(opt, ARRAY_SIZE(opt), "%s.%s=", - reg->name, reg->fields[f].name); + // append '=' to obtain '.=' + memcpy(opt + len, reg->fields[f].name, flen); + len += flen; + opt[len++] = '='; if (memcmp(cmdline, opt, len)) return -1; @@ -232,15 +233,21 @@ static int __init find_field(const char *cmdline, static void __init match_options(const char *cmdline) { + char opt[FTR_DESC_NAME_LEN + FTR_DESC_FIELD_LEN + 2]; int i; for (i = 0; i < ARRAY_SIZE(regs); i++) { const struct ftr_set_desc *reg = prel64_to_pointer(®s[i].reg_prel); struct arm64_ftr_override *override; + int len = strlen(reg->name); int f; override = prel64_to_pointer(®->override_prel); + // set opt[] to '.' + memcpy(opt, reg->name, len); + opt[len++] = '.'; + for (f = 0; reg->fields[f].name[0] != '\0'; f++) { u64 shift = reg->fields[f].shift; u64 width = reg->fields[f].width ?: 4; @@ -248,7 +255,7 @@ static void __init match_options(const char *cmdline) bool (*filter)(u64 val); u64 v; - if (find_field(cmdline, reg, f, &v)) + if (find_field(cmdline, opt, len, reg, f, &v)) continue; /* From patchwork Tue Sep 12 14:16:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381774 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9B003CA0EEC for ; Tue, 12 Sep 2023 14:18:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=v4KZ9iliifinFm/VPdOa5Qs97h0Ef8yP1+7O+/UlvTE=; b=wEuvSwHt5JrYK3acQ1KPDCJ1Jc L+BEJopAGo3H0+ZBL0VwdX5Zh72LoKg6F89YIRgjP54yAPUGilAfq7eSKm8fHDkOPglelwDrGRayp Aa2vmGtHiQjdt+j2wym+Ap9Lo7O1k1mtW6uwwRvZoUB32r+1yGGfoRjyX0KqGIMS7PszNhgUYNxSv 48Ug6UwrxO5JROrcLmpNE6PWZZdFPWRcl41PpfKZAKzSRskNzgHSMXDnrZrTH80GR92glwTgaRAS7 /XNy9zfItMISXjKXKca2rl5Ogroo/x3T+V+vZ8uCSXtyIgTqJNUPz9JAcpvRuFiCIMISjpSdUC+L+ RXBw9r4w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Dj-003Wdv-25; Tue, 12 Sep 2023 14:18:27 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DK-003WHt-06 for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:09 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-59b6083fa00so43633867b3.0 for ; Tue, 12 Sep 2023 07:18:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528280; x=1695133080; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=DZ6BAESjoTESHzBJpPMGEjJbYG4JorqLYEqotZx2wE4=; b=u+dcXGFFAHdjPR6MnYvcybGQ0nb8qjQ6UbBBDb/V1Axw4IxzJPj7LTnIsFB+lw/Lpl lA6eVH73ZTZwunTA4pXV/Wjia1nKoWWxNYA3Qx3hg70O534TcIqi1R/6fzGfxz5QzQ6l 4CTQxOiEn0tp0wNnlmvO1C5/gjmcpw2HzDM1FB4vxk5qx+U2/tDfLgbCuPZsV4jUiIQf 0Ahg/8t1OXS6zFVxWOd1z3C13g2oklPkUJRwaDcC8TXRSmC9TDkf7Hm3835x6CfRnRTo cIWrjkRJkrPBosBpGLrU3K9VjIPM8NTNGjmXGEvQLmbXobROZbzOQD70R52SxzQcZb0u 6WOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528280; x=1695133080; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DZ6BAESjoTESHzBJpPMGEjJbYG4JorqLYEqotZx2wE4=; b=vSEDSxrjtJ20t+g/oP3j5LNxzRB6r3EI0OQMyMLLI1SfXM+Dg/neTCBRbNqsOUUmsK LMXbNkvZmTU6xQQnLCn760UYCOV7uzSZLtT1YWl6Y4w8JM9Nuq+cl0h5H5evtV23kGZy xpnWFJjETStLSy1yR0Bbu0w70Q1K64WC48mE77W9Vxc3RoTXkvKCvs0IJRowtvXsXfTy 4utnn5DvBBxJJ07vmRpH4MlLrgCff5XtJm4IQqd9JkF3aQuqTAWz0CqFILite+Y2a++h 2Hkg/eLiMS9FzoUOAtIXUTDvqxj6bq4nYFsdmdCDNfTJqH1pFX0q9/4n61GgePHS6Crb VGzQ== X-Gm-Message-State: AOJu0YzgcuPYpba5nRiRZNHrit9t37IW1FG9Pbjv8w+aEG0PucjdO2jI uI7Mtuj3uq6+1a6H70N1Krq5t8ORcVO/MBKLpr+Gz52x2RmGcnyHlUjj6OZmC6npfNVqABYK2Dv bNTUJOs1k8bdLhqqzOTTqp1EsGyjnA59698vdbfA73k4WOQi0AcLgccL2zLCXnUhkaDSbtsGkCF I= X-Google-Smtp-Source: AGHT+IEh8UTm7zy1Zd7oKrQd44AqYWet4j2viCM2PKqcHgv4P8DzE4lg0eyWbDuzBAw1wvQw/twy/DiD X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:7642:0:b0:586:5d03:67c8 with SMTP id j2-20020a817642000000b005865d0367c8mr360381ywk.3.1694528280316; Tue, 12 Sep 2023 07:18:00 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:09 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1679; i=ardb@kernel.org; h=from:subject; bh=41N28jcvzRhByazbRSwqcfuI8Bs+Iu8HC5RqEulc/ck=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaNWLeDnJrG3cL4/kOJosfTAnpElZUfFHVgxTavZsx 6eL93B0lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgInwKzH8T19Xrf47ee21dV+3 JE/m2K/3Y5vGjEW/OebvS2w7tqMuUIWR4R5HrKB1MhfX6v61/15JvpnO1fuIV+Zwq8wSzReiTnZ buQE= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-82-ardb@google.com> Subject: [PATCH v4 19/61] arm64: idreg-override: Avoid kstrtou64() to parse a single hex digit From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071802_101968_14B20028 X-CRM114-Status: GOOD ( 16.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel All ID register value overrides are =0 with the exception of the nokaslr pseudo feature which uses =1. In order to remove the dependency on kstrtou64(), which is part of the core kernel and no longer usable once we move idreg-override into the early mini C runtime, let's just parse a single hex digit (with optional leading 0x) and set the output value accordingly. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index 41eeca857b32..2eb81795934a 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -215,6 +215,20 @@ static int __init parse_nokaslr(char *unused) } early_param("nokaslr", parse_nokaslr); +static int __init parse_hexdigit(const char *p, u64 *v) +{ + // skip "0x" if it comes next + if (p[0] == '0' && tolower(p[1]) == 'x') + p += 2; + + // check whether the RHS is a single hex digit + if (!isxdigit(p[0]) || (p[1] && !isspace(p[1]))) + return -EINVAL; + + *v = tolower(*p) - (isdigit(*p) ? '0' : 'a' - 10); + return 0; +} + static int __init find_field(const char *cmdline, char *opt, int len, const struct ftr_set_desc *reg, int f, u64 *v) { @@ -228,7 +242,7 @@ static int __init find_field(const char *cmdline, char *opt, int len, if (memcmp(cmdline, opt, len)) return -1; - return kstrtou64(cmdline + len, 0, v); + return parse_hexdigit(cmdline + len, v); } static void __init match_options(const char *cmdline) From patchwork Tue Sep 12 14:16:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381775 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D5D67CA0EEB for ; Tue, 12 Sep 2023 14:19:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=aOoVTAwbbjkNJmnxVBJEUf7g4/iQpBmlBcC4QdN112g=; b=qN9zVpLvEuxqCGB3QjW8HHiA70 /BV2Siq8jxv9GPEzR+/0tzRjLUCh4XeleU6OB/8tvof4863hLoS1dDuJZYLAqE7cvcd8CBo6WkiZ5 B4YYua2lj5vWXTM8K+2h9m7NLMl5xRrNMi2ywcqJEGKkmdTxFdmd39Ex6u2P/eAHhlCuuD4XRKPhq Qg3Mjpg6ujZ27KlgjnATAsybLkvn/UVdiCxQITCo7jk2QhrWEkdU9aRmjvt7UQrKa2kkta15MBV6E bWWCwuUrVbcKdOeoMEzFvE2cf7Wtfe58RgES4J7BzEIHw3El0cMa5KmjRn/4MEqTOXREbVMM85YJq 5Ro1YRRg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Dr-003WlJ-1u; Tue, 12 Sep 2023 14:18:35 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DM-003WKf-2F for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:13 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-d7e79ec07b4so5568955276.0 for ; Tue, 12 Sep 2023 07:18:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528283; x=1695133083; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0JbO4b5LAql/4zOWm/s3+C/nCX3E4nbsNz2NyZTVE9Y=; b=Ni0XPbGOzf1cYYgSISqiROke2RIsSpgQ/SOZpCE31Qy/xa/FqUV7LkRSAZYfx2qQtP vfJDfTMCpp+hGk04soAUNmYJwhpVTQT64WFOpJEosnMUITZMqCPyhiUPZuI6zWfWb507 7tutvuct5tc38Wh6EQ/OmJXflUlkVb1r/Wu4E8Ta28Al7QlK6KPvSQHRWZI25IXpzwOF ConGuojFy66WHrh35Vb/dOtRPbw8W21D7ad9FHSZppW5oelUsP/6TcBZ5AxKLYQ2akP0 wUwxW8jQP5uUwjtLbTap0DY1sXEHxuQQthOciYSrwatTQF1o+0txtkAPHiM3O6XSE0th 19ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528283; x=1695133083; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0JbO4b5LAql/4zOWm/s3+C/nCX3E4nbsNz2NyZTVE9Y=; b=NGTye2YG5x99hGnWkNKvaEvHFkxTvYD02Z1sGK/XGTmRg8z7e/KpJ7G+NThVwDxKnc Wx0YjbmojuvVYhu8F7Jeqkh+R5FIXFO2ZMwdfFJQNf95DDPI1MsPeL9TWE8ZFY4/55ls oJeq0gWhF+Lz5r5arnqex7O/pE92llkcCMjg4i15QhBcP64zF8JNEy5bsYfzfOEMiuNi 3xEx8zpBtPt/yVIbUdFB1ughZ9Xds9B/wYAJ/WwNVAvYllBbbAe41YoTwJpxcnYwD82s 6VplSqBTcfLq4ruReaogfCIznHV5yPw3beBj9MLJwkmAsB6dvqzlVA1OIWXHGckzDfsl z4wA== X-Gm-Message-State: AOJu0Yw6rzJ8RcLSM3xuGEHxtUYmFTJzxNfpnmMD9Xx65IR2fEfqftql Z9UPzvz6hruIjPBs2uxJDRBpQ41CfV9/IEV9G3ulkvJvdnr04GaN/xZRx5Na860wyi/i1OI1WWo QNHZq3vdf7dqzZ1Jm7iCuGZPLN+6ynQDVk/2W4ZYgGAI6mIw+QmJMXlBDWTyvwEDQUFelm9d48U s= X-Google-Smtp-Source: AGHT+IGnKZUJngvBNGEWFfWmRQD9r3dc+k9OEZiMx6U8RR/0u+KWdhWnHJNZBiSZYdvM8EDCV5jbZcAV X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:7485:0:b0:d78:45db:d890 with SMTP id p127-20020a257485000000b00d7845dbd890mr293668ybc.0.1694528282883; Tue, 12 Sep 2023 07:18:02 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:10 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6731; i=ardb@kernel.org; h=from:subject; bh=6ULh28xw1TNI9vhKmsljGUL5Y6LwDd3QlsaTbYqGJm0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaLXO873FwlWXZp/wSOjSmuK+JONZ4sYk7+7pDMVbP BkklJZ3lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgIk03WVk+GA26xPHtalHdDV5 +03k3BXWRry5+tVv7xE/1Rn98dFHwxn+Z2XNrD54hvvshgdNV7akMazSbnQ4PDmiy2IH3+T5FgU vmQA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-83-ardb@google.com> Subject: [PATCH v4 20/61] arm64: idreg-override: Move to early mini C runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071804_790962_21111790 X-CRM114-Status: GOOD ( 18.58 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We will want to parse the ID register overrides even earlier, so that we can take them into account before creating the kernel mapping. So migrate the code and make it work in the context of the early C runtime. We will move the invocation to an earlier stage in a subsequent patch. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/Makefile | 4 +-- arch/arm64/kernel/head.S | 5 ++- arch/arm64/kernel/image-vars.h | 9 +++++ arch/arm64/kernel/pi/Makefile | 5 +-- arch/arm64/kernel/{ => pi}/idreg-override.c | 38 ++++++++------------ 5 files changed, 30 insertions(+), 31 deletions(-) diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index a8487749cabd..dc85dc2ee4ed 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -33,8 +33,7 @@ obj-y := debug-monitors.o entry.o irq.o fpsimd.o \ return_address.o cpuinfo.o cpu_errata.o \ cpufeature.o alternative.o cacheinfo.o \ smp.o smp_spin_table.o topology.o smccc-call.o \ - syscall.o proton-pack.o idreg-override.o idle.o \ - patching.o + syscall.o proton-pack.o idle.o patching.o pi/ obj-$(CONFIG_COMPAT) += sys32.o signal32.o \ sys_compat.o @@ -57,7 +56,6 @@ obj-$(CONFIG_ACPI) += acpi.o obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL) += acpi_parking_protocol.o obj-$(CONFIG_PARAVIRT) += paravirt.o -obj-$(CONFIG_RELOCATABLE) += pi/ obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_ELF_CORE) += elfcore.o diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index a8fa64fc30d7..ca5e5fbefcd3 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -510,10 +510,9 @@ SYM_FUNC_START_LOCAL(__primary_switched) #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif - mov x0, x21 // pass FDT address in x0 - bl early_fdt_map // Try mapping the FDT early mov x0, x20 // pass the full boot status - bl init_feature_override // Parse cpu feature overrides + mov x1, x22 // pass the low FDT mapping + bl __pi_init_feature_override // Parse cpu feature overrides #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS bl scs_patch_vmlinux #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 2cc3aa4c27c7..011a9dd93bd2 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -35,6 +35,15 @@ PROVIDE(__pi___memmove = __pi_memmove); PROVIDE(__pi___memset = __pi_memset); PROVIDE(__pi_vabits_actual = vabits_actual); +PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); +PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); +PROVIDE(__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); +PROVIDE(__pi_id_aa64pfr0_override = id_aa64pfr0_override); +PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override); +PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); +PROVIDE(__pi_id_aa64zfr0_override = id_aa64zfr0_override); +PROVIDE(__pi_arm64_sw_feature_override = arm64_sw_feature_override); +PROVIDE(__pi__ctype = _ctype); #ifdef CONFIG_KVM diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index d084c1dcf416..7f6dfce893c3 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,6 +38,7 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := relocate.pi.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +obj-y := idreg-override.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +obj-$(CONFIG_RELOCATABLE) += relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c similarity index 94% rename from arch/arm64/kernel/idreg-override.c rename to arch/arm64/kernel/pi/idreg-override.c index 2eb81795934a..bcba0ce71af0 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -14,6 +14,8 @@ #include #include +#include "pi.h" + #define FTR_DESC_NAME_LEN 20 #define FTR_DESC_FIELD_LEN 10 #define FTR_ALIAS_NAME_LEN 30 @@ -21,18 +23,6 @@ static u64 __boot_status __initdata; -// temporary __prel64 related definitions -// to be removed when this code is moved under pi/ - -#define __prel64_initconst __initconst - -typedef void *prel64_t; - -static void *prel64_to_pointer(const prel64_t *p) -{ - return *p; -} - struct ftr_set_desc { char name[FTR_DESC_NAME_LEN]; union { @@ -329,16 +319,11 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) } while (1); } -static __init const u8 *get_bootargs_cmdline(void) +static __init const u8 *get_bootargs_cmdline(const void *fdt) { const u8 *prop; - void *fdt; int node; - fdt = get_early_fdt_ptr(); - if (!fdt) - return NULL; - node = fdt_path_offset(fdt, "/chosen"); if (node < 0) return NULL; @@ -350,9 +335,9 @@ static __init const u8 *get_bootargs_cmdline(void) return strlen(prop) ? prop : NULL; } -static __init void parse_cmdline(void) +static __init void parse_cmdline(const void *fdt) { - const u8 *prop = get_bootargs_cmdline(); + const u8 *prop = get_bootargs_cmdline(fdt); if (IS_ENABLED(CONFIG_CMDLINE_FORCE) || !prop) __parse_cmdline(CONFIG_CMDLINE, true); @@ -362,9 +347,9 @@ static __init void parse_cmdline(void) } /* Keep checkers quiet */ -void init_feature_override(u64 boot_status); +void init_feature_override(u64 boot_status, const void *fdt); -asmlinkage void __init init_feature_override(u64 boot_status) +asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) { struct arm64_ftr_override *override; const struct ftr_set_desc *reg; @@ -380,7 +365,7 @@ asmlinkage void __init init_feature_override(u64 boot_status) __boot_status = boot_status; - parse_cmdline(); + parse_cmdline(fdt); for (i = 0; i < ARRAY_SIZE(regs); i++) { reg = prel64_to_pointer(®s[i].reg_prel); @@ -389,3 +374,10 @@ asmlinkage void __init init_feature_override(u64 boot_status) (unsigned long)(override + 1)); } } + +char * __init skip_spaces(const char *str) +{ + while (isspace(*str)) + ++str; + return (char *)str; +} From patchwork Tue Sep 12 14:16:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381776 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0389BCA0EEC for ; Tue, 12 Sep 2023 14:19:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=xS08Y3+goom75x2crzsAwpG1z6h0aK6J0CaEBmHRnlw=; b=YpbuhG+rixeT0RXaETUpOIiGkw W6YbD9QF001Y93r8TiAQd+Q/nQ/V1DUSQRRzcs0tw9fXuylu6aNNqkinnHKkYWDSDuPfyQUanRDGT S/EPERohZc2T2BJ786N/QMnQvDjWpPNyZURzPZooKmXYeL8oSX3SxFYRXOGaxQD17y9wWUorUhwd7 AEY5UsIMhZgL1LJoiGHZo/nw1zGdV+JVD3ANWrKDklr1SZ78yF7jUOMsYeDcsXJNNb0a5uINHWcGU ixaIy3KmNgSgmP4Cn9IOTGj4jMZ9WKJrSlvpcGzzU8LkTz352UWekW6aeHFllEpkY4hkKLg+OMkKX vvps7Dgw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Dv-003Wok-17; Tue, 12 Sep 2023 14:18:39 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DQ-003WMc-01 for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:17 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-31f9186eb8dso2019220f8f.1 for ; Tue, 12 Sep 2023 07:18:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528285; x=1695133085; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NQhu/6aj13Dt2lH4DoK1t46t8648KCj8JbD+ZI51H4w=; b=z7XukCY/SYghLUpN2U2EOd5gh0eWtMzi6ZlkxRPUI0dLZC/o6zF1hz/YoTBtASjz95 qDX9apcRY3bWRtARtIRmogiVkEEzSYVrj/izjZA6VbM9aS+PT8fHvVGse9rggJLRdjE8 n8MXa3vNPoyfJ2hXhxLkMz+1QXqjdExKp26/SKdJL0IGtTWdB73rlQP9MEWEIglBvz28 VksmqAG6ZcdgblZJs2lbt5AAVlF52LFjfMfQ4Zkv1o2J/Vdl2QMCcjxDimMjZdPaZ89E c+OwVj1BHd44wS9PI/8J7crus4Gkhuj4uwfyuIeXfozIuTkebCQbaUTTRrhBwORU0nMH Qj7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528285; x=1695133085; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NQhu/6aj13Dt2lH4DoK1t46t8648KCj8JbD+ZI51H4w=; b=D9Eh5Zdp/tdMmxlpX/uSnZZI7+2K7Qu0LEpIAgUHpgdl1TNkJj6hzzVfu/qaB+fyFe VW0LceA7IPGlPUCdYDFm4yS138AaS5MoXWN+ACC58AVekAWuSu+i0Cw+qH0YEmMsNDQI jXbfubJwrFC/Y2gLvVVsd7hQUk2iF2qIm6BthY4+GsizGF1FsPbZHJEL81/l0tvnUrYb 54NeLY+vKcGk+eoMpfB5LeOqMLyLhY9Li3jmyF4QLrn3gfPSPgJ6Yrfs9IZBzA84OmR2 kyx0RrnPwnHFoqSQgEp2Ga/hwyX3oR+ucqdvK2WBFh0BjOyb/ODkJXvgkD+PtJ6guC1E st/g== X-Gm-Message-State: AOJu0YyXsuDgXZFUAjbh2zd/SzVYutgGiOSum0S01P3E6gcy6nJBbFSm j3mcNRwHk6zLTJPFvUCIiaJTRnqGjn7lh0IDBQaf4tJ5f4hrkYpzwszjHHkpDs0kglvhZ0eaS+L 0wM0wBfoE9kQSJgpm7W6Ddtv92e7Ob8myDM3uLiMTL2jzGjB+oR3ejMyqGHtoNTfhAp4MgWQ6ji k= X-Google-Smtp-Source: AGHT+IHVF4WomTKZ/KFxSQKvMpnGbm287loEaUqNK7vgAglKJPH3ZjMv0bPF8TPIpyTzuvL3QN68omKo X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:f546:0:b0:31c:6800:c4eb with SMTP id j6-20020adff546000000b0031c6800c4ebmr146279wrp.13.1694528285219; Tue, 12 Sep 2023 07:18:05 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:11 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1354; i=ardb@kernel.org; h=from:subject; bh=Rs4Isya5F550+UA7pfxX/X5G/PKtnTuW3YNEIEX/nnI=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaM0cN1Hl/XICT7TW75vz55S5rr1iRbLN0wh+LxGh/ 3svzg/rKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABNxdGJkaF++tkdfe9LsNRE5 vxnbM7NjV29katx4U6rMKlrg3zHdtYwMk3LvyLgZTrqREuZa63+kTvQtewJbW+3xKBfDxbdKbvq xAwA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-84-ardb@google.com> Subject: [PATCH v4 21/61] arm64: kernel: Remove early fdt remap code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071808_082951_06BF1308 X-CRM114-Status: GOOD ( 11.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The early FDT remap code is no longer used so let's drop it. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/setup.h | 3 --- arch/arm64/kernel/setup.c | 15 --------------- 2 files changed, 18 deletions(-) diff --git a/arch/arm64/include/asm/setup.h b/arch/arm64/include/asm/setup.h index f4af547ef54c..acc5e00bf3b0 100644 --- a/arch/arm64/include/asm/setup.h +++ b/arch/arm64/include/asm/setup.h @@ -7,9 +7,6 @@ #include -void *get_early_fdt_ptr(void); -void early_fdt_map(u64 dt_phys); - /* * These two variables are used in the head.S file. */ diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 417a8a86b2db..4b0b3515ee20 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -166,21 +166,6 @@ static void __init smp_build_mpidr_hash(void) pr_warn("Large number of MPIDR hash buckets detected\n"); } -static void *early_fdt_ptr __initdata; - -void __init *get_early_fdt_ptr(void) -{ - return early_fdt_ptr; -} - -asmlinkage void __init early_fdt_map(u64 dt_phys) -{ - int fdt_size; - - early_fixmap_init(); - early_fdt_ptr = fixmap_remap_fdt(dt_phys, &fdt_size, PAGE_KERNEL); -} - static void __init setup_machine_fdt(phys_addr_t dt_phys) { int size; From patchwork Tue Sep 12 14:16:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381865 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 930E1CA0EEB for ; Tue, 12 Sep 2023 15:30:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=sQ5KFxDoeLSPICqaOJgGyzNgVvXYCEQuGcedzk716dw=; b=v8AJaL7UPXAUhy6gF3wiH8I+hI nTBqLJV6xXuRqSLa+XIgjsF+L/0nw+u7MxV4ZmqvpBKVLZgB18fHHtQwW8Q/Q/93qLM1R0iw/Fu8s KrOaUxxxlhd9PviPWPAClGwZsDbTZL75OhSF114rRA1/xp8vMGpuzHf+t4uvzlnAcyXBgGdm6e9tA GvwI2TfITYc5KcQpk0g/7HrEgCdKZ+NW9fiV/pleQdTqjYSZXC9TQ8PQw7HcQZXiun7qLzzYifMM7 vMBlKVnP/XdQD4RrB2AhR3+gE8bkAi+j9JmcylODR3GOTdNgu10JRc+6wCIjItbxuWXwaoBBfsrK3 e87zGdHA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg5LP-003igH-1j; Tue, 12 Sep 2023 15:30:27 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DR-003WOt-1n for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:19 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-59222a14ee1so67432837b3.1 for ; Tue, 12 Sep 2023 07:18:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528288; x=1695133088; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7zFAMdEnBh2nxO6Ud4c95SHScST5+qYd/avAx/j2osg=; b=oVKYmtPS5MWzOEkpIo1bHxVdg1DFPFneHXguNnL2hQPs723OQPBcOFyvKgLqY0Lsll Pe7R8aCD3pw+zlSWLOeCC0BDp8Cleih8yWME6VorgvevyliOGu2DF6BPySb9dTFsha+5 zC6qmeVXxHBXHHDTOJ6YDrcc7zRAmcc0GiAV9UEyh2LW7EiuCI+8CgiZMiLGKzC+eBMB 7dtZFPL7qgBQA1oz7uSiXKF1um2XSgYVW/44iMktLj0P/Gozwe3z4nrolWQp1MwnEOav wRbmrvTrUGZ7GB9vEAmeVgvL5AJyqPg2MGIMNOcXZYr/7NLGoKzGO/zseu+XqNdBPaIg Vpag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528288; x=1695133088; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7zFAMdEnBh2nxO6Ud4c95SHScST5+qYd/avAx/j2osg=; b=VdHcoF10N5l+m/76Tj5GQ/qb3w8tpb7aTboiyVoLEq8pXW6HwwPXUsivhm/YQOh3tA G/XSPulZyfsd1Lv6J9G3c0eF8xKgYRAxo16OzQBaZvP25dS8OyFW0lrNn23fwEKqqWwf O9hX5Z9JrRX/g24J/2B6+lMizdfb09DT+slv6TUSIlbZuTN+ZfV4CYtvcRtIqhsvkl3C J4u/zCe0KYX/HV8NvuU6SbtePXgLZN/XBAkMVT/8a6Fm9Z4E8g0E87Jtgl7SefLuG83u ex3okVQz/cmFIMTcdcXoV8YzLd4VoG94jIqd8ubzvlyApnPtBHmSj5GwZaShPPiYIQum rU6g== X-Gm-Message-State: AOJu0YwG1zuBKF1pfTY5rHChZwDm64sz9erxFOlSCrw62QolFp6DLRF6 viggxFImdtEjdwseRqKlNsWutOFuqM+VjQcC9w/X1OrTlobOE9cwOpHwZLCR1scPd2W5+vrWmw4 fFt42NKcbNoUH4sBkFwS1jSfKoq6v5AV4s+E9IOyAdo6ySTkZY0hWaERdl5dJ0zX8r0m7dZyBeS 8= X-Google-Smtp-Source: AGHT+IEJ1RparPTsqiyFdzyxH9A0rL3Ams+gFuAfWW7e8BaoLYZJKNK9N6KCsbIowMZuPk0juaqSCp4F X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:4327:0:b0:589:a5c6:4a8e with SMTP id q39-20020a814327000000b00589a5c64a8emr309653ywa.1.1694528287760; Tue, 12 Sep 2023 07:18:07 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:12 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3389; i=ardb@kernel.org; h=from:subject; bh=6Jap1NIlQWA20VBoBeawuJQlFyCJAj1ChLFh6mmy7So=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaG3vmtOSlzUXrNj/r2avquSHnowPCgdvPT/eOcNp7 sG/C//97ShlYRDjYJAVU2QRmP333c7TE6VqnWfJwsxhZQIZwsDFKQATYRBhZJi97oTyf6Op3L2/ 3fY9ffXykzeXy7ZJ+6enhOVNmRbwdmMAI8MCzZ3KUzgnbF8u8Fhv79X/XQ8iD8TNKGpJ+FEU4mH 4cA0/AA== X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-85-ardb@google.com> Subject: [PATCH v4 22/61] arm64: head: Clear BSS and the kernel page tables in one go From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071809_613530_5259F28B X-CRM114-Status: GOOD ( 16.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We will move the CPU feature overrides into BSS in a subsequent patch, and this requires that BSS is zeroed before the feature override detection code runs. So let's map BSS read-write in the ID map, and zero it via this mapping. Since the kernel page tables are right next to it, and also zeroed via the ID map, let's drop the separate clear_page_tables() function, and just zero everything in one go. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 33 +++++++------------- arch/arm64/kernel/vmlinux.lds.S | 3 ++ 2 files changed, 14 insertions(+), 22 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index ca5e5fbefcd3..2af518161f3a 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -177,17 +177,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) ret SYM_CODE_END(preserve_boot_args) -SYM_FUNC_START_LOCAL(clear_page_tables) - /* - * Clear the init page tables. - */ - adrp x0, init_pg_dir - adrp x1, init_pg_end - sub x2, x1, x0 - mov x1, xzr - b __pi_memset // tail call -SYM_FUNC_END(clear_page_tables) - /* * Macro to populate page table entries, these entries can be pointers to the next level * or last level entries pointing to physical memory. @@ -386,9 +375,9 @@ SYM_FUNC_START_LOCAL(create_idmap) map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT - /* Remap the kernel page tables r/w in the ID map */ + /* Remap BSS and the kernel page tables r/w in the ID map */ adrp x1, _text - adrp x2, init_pg_dir + adrp x2, __bss_start adrp x3, _end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 mov_q x5, SWAPPER_RW_MMUFLAGS @@ -489,14 +478,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) mov x0, x20 bl set_cpu_boot_mode_flag - // Clear BSS - adr_l x0, __bss_start - mov x1, xzr - adr_l x2, __bss_stop - sub x2, x2, x0 - bl __pi_memset - dsb ishst // Make zero page visible to PTW - #if VA_BITS > 48 adr_l x8, vabits_actual // Set this early so KASAN early init str x25, [x8] // ... observes the correct value @@ -782,6 +763,15 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu + + // Clear BSS + adrp x0, __bss_start + mov x1, xzr + adrp x2, init_pg_end + sub x2, x2, x0 + bl __pi_memset + dsb ishst // Make zero page visible to PTW + #ifdef CONFIG_RELOCATABLE adrp x23, KERNEL_START and x23, x23, MIN_KIMG_ALIGN - 1 @@ -796,7 +786,6 @@ SYM_FUNC_START_LOCAL(__primary_switch) orr x23, x23, x0 // record kernel offset #endif #endif - bl clear_page_tables bl create_kernel_mapping adrp x1, init_pg_dir diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 8dd5dda66f7c..8a3c6aacc355 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -311,12 +311,15 @@ SECTIONS __pecoff_data_rawsize = ABSOLUTE(. - __initdata_begin); _edata = .; + /* start of zero-init region */ BSS_SECTION(SBSS_ALIGN, 0, 0) . = ALIGN(PAGE_SIZE); init_pg_dir = .; . += INIT_DIR_SIZE; init_pg_end = .; + /* end of zero-init region */ + #ifdef CONFIG_RELOCATABLE . += SZ_4K; /* stack for the early relocation code */ early_init_stack = .; From patchwork Tue Sep 12 14:16:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381777 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9ABC3CA0EEB for ; Tue, 12 Sep 2023 14:19:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=+zxqpnQfEV9ewWDK/tBTU/Ii4iIS9fn1mbTpp+fLbLI=; b=s/PX1tX3cNU7mO3VohNPixPsNV Tk5k2aN7pMoaVH5WxxKK+ITgJk1IbPOYIuvDadZaj928Cde583urQNP2yYupX8L8/6nOGSvi4CRQx 7w10bdcdX5Jlr+pVtGm+86dHFjIDeYC65zyveP9lbY4jGIF2Y3PUD9zoBih5oQrxYwnAX6/EeTroX XRPY5Ae3D6TY2+bsmI9IlbQ/XEWAKUXjyTybmq/hvZRPkOCisYal++uz16/1US3GhCrZgBxcmcOlN vyYC+CCDfVFLri7jEI9HTxfVsQ8z/EG7F9flsxuMONIciwzqNHgT9nSJE9d9sha95khzwZ38GDw0V B7A9/FEQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4E1-003WtP-0I; Tue, 12 Sep 2023 14:18:45 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DT-003WR0-3A for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:22 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-d8011973047so4416088276.3 for ; Tue, 12 Sep 2023 07:18:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528290; x=1695133090; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BGLA3neG+tb8jih86vno7Pk9eHTdG9q4j4NBkyiMcC0=; b=aTR5djViDm96cxObW/3Z/IOAoiV81PhBB8CN6w7o8zWPrX3r/rQg3GBlvDRluuyO4G NZ7AtFKSIp8P3Kq2KtcScyTaTGp+W36ueKLQQa5yD8qfsiqRWhXmfxA1tThRLb7isgu5 vf0gtA61LX2SwRsaeFp5bbb1KCrnAKCFbn/SGwCOSi9HYXYe6Ss2ivZxmCw2CdmJZORY lNL7D+s9OHutUOVLX1/eK1yaz2FpdJ3wBcj9M5Bv7MRlzDV339MFnPvYA76gkGGg+7rn hNzVpYJn0O3DSC15mc0Q3mrcY00sSK3odX1zGsmcmdYx2nKUngVx+92PsvtQOxC0t9jX cc2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528290; x=1695133090; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BGLA3neG+tb8jih86vno7Pk9eHTdG9q4j4NBkyiMcC0=; b=VjaUW2iB67b2oB2ICVe0XqSE0vln0t1CPSAjdBcb3cdfAu8dMtk0gr2Ag3Mnk4tHPM q5s3DO8LGBnvOUc638CWeiw1GXkAA5Ozjsvv5rkANlfhBAc8AUubmkbqdEiG6YiWORNA VgPqqpzfJyZnzGIQo+g4EhkniKb6Ckd3jl8omDzaVNghdaGa4KaajLt0UBE+uACuU7kK vxs1z/FwgDK8TdP073Zg0djQpVXaCX5WrsgO2rojoAUilm5INIp/clq9P4e3ZGKffqSa OiiKB3/3fO7hzqcZJ35lfAhYwQ1a7ZZ977bVejSy+Ko9ylW9AsloAqxkKkrSbRes9rxT Qd1A== X-Gm-Message-State: AOJu0YyV5Isyplx8ZTvmbH6PxmeNaZiU2vaQbvmzuAzInqWkjedubrOr iTmDK5McyyZpwISwrLZWRjdax8CPfufcj/WGKLnCz+kqA1eDG7sTXhZ3Rt18OhTchW0d1u/dlqX 2Hr9u5u4ozKAvHVEWp8QsFeAHT1S+J4GUFh1u2WVNYKySxFL6CiTVHe+jfxrcTXIIK7psFqnbhZ A= X-Google-Smtp-Source: AGHT+IEp18tK5oHipL8CLmugXbiEAd2+J6a9yLjciixnrVi+Ql1ysySd/IDjqwMAIpI4vN6iS6pK3Sal X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:ae96:0:b0:c78:c530:6345 with SMTP id b22-20020a25ae96000000b00c78c5306345mr249032ybj.7.1694528290167; Tue, 12 Sep 2023 07:18:10 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:13 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1677; i=ardb@kernel.org; h=from:subject; bh=cSiuqU58eWTFUnEgW7cbqc89QaTx9/6Ur7/IQsEMtvY=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaN29ktnfT++4d+1pDXeYkI//r68r5H8XFh9TqkqxF 171K721o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzkDR/D//yLbiqTNYxO7rsp uqD+Yt+6Q0caOCX5M/9NfHTFsfBGjCXD/4yPk3mtJl0qOrb9YIul+6vgtYJnsi/3rpQqf+w7t33 xMS4A X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-86-ardb@google.com> Subject: [PATCH v4 23/61] arm64: Move feature overrides into the BSS section From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071812_062647_8EB3A1CC X-CRM114-Status: GOOD ( 12.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In order to allow the CPU feature override detection code to run even earlier, move the feature override global variables into BSS, which is the only part of the static kernel image that is mapped read-write in the initial ID map. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/cpufeature.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index b018ae12ff5f..bdd492cea88b 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -664,13 +664,13 @@ static const struct arm64_ftr_bits ftr_raz[] = { #define ARM64_FTR_REG(id, table) \ __ARM64_FTR_REG_OVERRIDE(#id, id, table, &no_override) -struct arm64_ftr_override __ro_after_init id_aa64mmfr1_override; -struct arm64_ftr_override __ro_after_init id_aa64pfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64pfr1_override; -struct arm64_ftr_override __ro_after_init id_aa64zfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64smfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64isar1_override; -struct arm64_ftr_override __ro_after_init id_aa64isar2_override; +struct arm64_ftr_override id_aa64mmfr1_override; +struct arm64_ftr_override id_aa64pfr0_override; +struct arm64_ftr_override id_aa64pfr1_override; +struct arm64_ftr_override id_aa64zfr0_override; +struct arm64_ftr_override id_aa64smfr0_override; +struct arm64_ftr_override id_aa64isar1_override; +struct arm64_ftr_override id_aa64isar2_override; struct arm64_ftr_override arm64_sw_feature_override; From patchwork Tue Sep 12 14:16:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B8020CA0EEE for ; Tue, 12 Sep 2023 14:19:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ak9ngEBvTbT6oevxb0XUcoG1DOKCLSrZl/ZAylEW4PA=; b=aS0MDrqL6YEt1WvjIvj/sfycg2 G5BD/i1M7QEoP0Wf81LIpIWWpGIVj7MXNDaYbue/zZgmPBzOQStntCfT+AR6gyKTs/4itelUDbZwI Gbcp7wXsokIV7MpoTh6HWZeF+J+AGWmSX2cFpupDqYQShWhyHa0HK2SOz3vpn5Na9dipTd7kk6+BQ +fTMYawGYWF/ZhAt63Pu2RzEIqPMWsXPs8KPudZAdr3fheybGSOZX7xz++NndZnLV6GKVUEAMERTZ GbRIBfq8IoCmmX4pnx/0p65ek9/Av2P26pJULBxc4opF2A7nQJ+vQQryAzTC6aAzSxpGnRZXKw6fU CTreiPyw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4E3-003Wvd-2l; Tue, 12 Sep 2023 14:18:47 +0000 Received: from mail-wr1-f73.google.com ([209.85.221.73]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DX-003WT3-2T for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:26 +0000 Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-2f2981b8364so3751685f8f.1 for ; Tue, 12 Sep 2023 07:18:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528292; x=1695133092; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qCRUv4C5mP+SXLet0v8nI0HgZvtJcNWosvDQvUdODoE=; b=S16SCKDghUia4y3mqCHHEtAmWwFDYe5AeaYrBPEqf7lFBS0epRq87XH9rTQ5GuMssm WKjo1RjRW/b9ODjxvaNG5slGIlVBTbD9KTASIp3KGScSU7j2qkivTYxR2Goa3+GXiB0I mV/zEWCLu6stjpQYVxtMM3/jEacp4PSqVnvFiyxGwMqYVoqeL/2vsaqETKgweY0YslUS hT2KYLO2KUJ2QQauq1DDNAv1bZjLQ3XlCw7/fLmHXDQoO2PX77LP6bRAYCJPmxEWWveR bIekSekdMvp+a1VqrjjecQ44gHxFQQgovWmFFarMwn0xgjtjvIQwZMwrtD+9b8y/4Sel 7npw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528292; x=1695133092; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qCRUv4C5mP+SXLet0v8nI0HgZvtJcNWosvDQvUdODoE=; b=BKBieXAQ+t7JrhewNmcEmws592d9+XS388CZvVe9Kr2n/QE5wyxp1RYsXrQIoyGN8W +WhIm0T78IU0CoNc4IGQakFJSuHS6Itrq0NvOFEP+cOg9Heq+WWFKZkRio7z0WUc+akt jwi4ZMXuWU49QfsIsc4ojLbUKTcsGnAwPkO2uwijJGqKzq/AfNZveXNlckk1xsT4CWkP /aDrzR+BgIO/dISo25qv11bKsJK+IisDA3HC7qnJgqWO1NH4zlNDVD6sdNxJPhY1hAu5 wfVf/p3laUeMLws/5IVoMm5YfWDnl8Jcy5b37GvJQ26rCcowA0UJ/HFBIwyg9mygy2If CNMg== X-Gm-Message-State: AOJu0YyelSfzk/B7KV8cc7wi9q3CTqicG4fd4P/iNy4NBUDRQNvlHn/2 WDjinyfawK+TYL2DToL7Tn/TmpiY7uET+lPMRcEs+4yrZXuQQ/+/NNjTGcfjRhb0y6u/3zDCv1y kBfNT8Z3rwDg1dp+mJTjs5BdDsRi8t7Uaa2fvhalO/rSNHTQCJR9vMd076sdojhSGVBoNEWDJuu g= X-Google-Smtp-Source: AGHT+IGNLrBuW67ESmWraKM7G5DRQZjq5fQx9yU4SA7Ob9tjvMVyihpaXQntC9z+b+sH+XsvOrImV/ZQ X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:f14f:0:b0:31a:dba9:ecfc with SMTP id y15-20020adff14f000000b0031adba9ecfcmr147176wro.8.1694528292421; Tue, 12 Sep 2023 07:18:12 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:14 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2825; i=ardb@kernel.org; h=from:subject; bh=0+stDv15msuniA5scr4PQhCtluP2XOOD+pMGFMh/LVQ=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaH3yk3rraxvNef4sVW2K//Nj1dyEXddyPP3q5rPYC K4qnWzcUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACaSrMfI8Gzr7T9xZofWbWWN t6nS+1H7rjbd7ky3vUCaqdKl9xueJzP8j4suuNI0/6rVJLuYnvWrT94Nb9a3m337slKjl1Z0uNJ 1DgA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-87-ardb@google.com> Subject: [PATCH v4 24/61] arm64: head: Run feature override detection before mapping the kernel From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071815_846163_552F1A5D X-CRM114-Status: GOOD ( 15.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel To permit the feature overrides to be taken into account before the KASLR init code runs and the kernel mapping is created, move the detection code to an earlier stage in the boot. In a subsequent patch, this will be taken advantage of by merging the preliminary and permanent mappings of the kernel text and data into a single one that gets created and relocated before start_kernel() is called. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 17 +++++++++-------- arch/arm64/kernel/vmlinux.lds.S | 4 +--- 2 files changed, 10 insertions(+), 11 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 2af518161f3a..865ecc1f8255 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -375,9 +375,9 @@ SYM_FUNC_START_LOCAL(create_idmap) map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT - /* Remap BSS and the kernel page tables r/w in the ID map */ + /* Remap [.init].data, BSS and the kernel page tables r/w in the ID map */ adrp x1, _text - adrp x2, __bss_start + adrp x2, __initdata_begin adrp x3, _end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 mov_q x5, SWAPPER_RW_MMUFLAGS @@ -491,9 +491,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif - mov x0, x20 // pass the full boot status - mov x1, x22 // pass the low FDT mapping - bl __pi_init_feature_override // Parse cpu feature overrides #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS bl scs_patch_vmlinux #endif @@ -772,12 +769,16 @@ SYM_FUNC_START_LOCAL(__primary_switch) bl __pi_memset dsb ishst // Make zero page visible to PTW -#ifdef CONFIG_RELOCATABLE - adrp x23, KERNEL_START - and x23, x23, MIN_KIMG_ALIGN - 1 adrp x1, early_init_stack mov sp, x1 mov x29, xzr + mov x0, x20 // pass the full boot status + mov x1, x22 // pass the low FDT mapping + bl __pi_init_feature_override // Parse cpu feature overrides + +#ifdef CONFIG_RELOCATABLE + adrp x23, KERNEL_START + and x23, x23, MIN_KIMG_ALIGN - 1 #ifdef CONFIG_RANDOMIZE_BASE mov x0, x22 bl __pi_kaslr_early_init diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 8a3c6aacc355..3afb4223a5e8 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -320,10 +320,8 @@ SECTIONS init_pg_end = .; /* end of zero-init region */ -#ifdef CONFIG_RELOCATABLE - . += SZ_4K; /* stack for the early relocation code */ + . += SZ_4K; /* stack for the early C runtime */ early_init_stack = .; -#endif . = ALIGN(SEGMENT_ALIGN); __pecoff_data_size = ABSOLUTE(. - __initdata_begin); From patchwork Tue Sep 12 14:16:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381778 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C9077CA0EED for ; Tue, 12 Sep 2023 14:19:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=g2Dt+nWaKxl+Bi2TqB2OzuFK9dTH4Q9NoEC+bizmy8Q=; b=xm6RpmM/cFkpWpzg2oeF+d8PD8 mxBgDZdSv3YPHkqNL+EH1LYbfeBWj/WZf9fjbeNjeSVeyXL29R+nVAQFBgQnsGgpMrIORWylfbdwU K00P55oH6xyzyFlWq8AswhJPV60Xlv7tbtUgVn7JzzybB4iTPf2cDC7s6He3PrEZpcvyuEpHpkctw F6qFMOorhRlYrWQ25L7LzynRYKZelN80p1jQB7+naT8Rh1PDCGWY0o9ReT4mHJPlc303zSyx9LtGp ictrLyTDVEAEADeX8pA0XX9Uavmu+RPLFdGPGCTSP+Y8giwmvYF5D6G3MzUDarxjK1JvxIK+jMsmG QYTrWwSw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4E9-003X0f-1c; Tue, 12 Sep 2023 14:18:53 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4DZ-003WV2-0v for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:30 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-d80db590b1cso3198565276.0 for ; Tue, 12 Sep 2023 07:18:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528295; x=1695133095; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MJYd3lTCvsjOLSBORSzk2v4/+nDoBR/U9Sn4XhI4spM=; b=AgFj6uIQocpUNc+vBi/NrCzkdBYfqOIQaFxRlSMRlo78sO6xsf+L5sIE1LGJJyFUZ9 N0oWWuQlnw7l2eYULF/1ciPGlKgpJ9dOctSItJopoeuii6UC+PHxEwShSIVvhErFYYrF 5M1ZBlifBPkQXIBHpoppdi8/KQuI4bcR28X+Sx7kb33rMHIQjKMp5B1w5APYy5C4FTvo 9um7U0PqKTF1I1T4U3tpuSScDSTd3D1az1+8oP132BqbIIwwVcQPM/4MdJBmm9ZojJ94 v7JHJLnMiOzK2azv8yXjGQknLTllunxtMnwETTFhlYk4FxYyFUjRUaz1qEo+fny81vrx 2wFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528295; x=1695133095; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MJYd3lTCvsjOLSBORSzk2v4/+nDoBR/U9Sn4XhI4spM=; b=c2yMKd4ZNmOWU8oqsqePB5oUGpnkxVLUO76sHH8iwHFCL+lAN3+CyAdRlVdTjFH8b+ kKz24sB8IWNoGEUU3JvzSFOhHaUYgu2fZWlfvJf4riHg0tKOhpmGv3pdVBpUuFlsJhan iwOlNv6ooWs/kOePfUZkt3I1UO9GASM07Yp9R6WTPn14i3dv3QbvJfV7hsy6L7kHoHbu vD1St8VPjds6QDeKUKi23XNNc7YnsAJf0V777kOW/2gNd6VfXTY383XlemBkrsax1rdi QDNtRxZ0kF7UP2c4myxPRecLCJjcH43hnD7ZnwJvX0agDvaKU7M5BCJlV2tI7WpOLJYw 413Q== X-Gm-Message-State: AOJu0YzsTXl9lNU5c04TKchRtFOnx6yyRmaocoMJPg9QSiCJRxhex+b5 PTX+SMSLfxYxVxNw+8nmtVLOnrHps7T+ZHqS5qI7TLxG4wZIEEy2D9dKhOF56GMBmV6WueOAwDH /y2Ulwt9xtmzV7KPOZSFAI0APYmTznECvAJMfEelnc5WFXqKWRqpmENS3MVe32xj671k0Ga7gql A= X-Google-Smtp-Source: AGHT+IFJM+NP3uZBHi/VU0uc2xx/SdEc5SUci3GVYb3SZgPn/X+1aPrQKoYOMNBMiCFUOCMVibIkPZ8/ X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:fb08:0:b0:d77:f789:f9da with SMTP id j8-20020a25fb08000000b00d77f789f9damr78300ybe.6.1694528295365; Tue, 12 Sep 2023 07:18:15 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:15 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6303; i=ardb@kernel.org; h=from:subject; bh=g5GBUcsqZAcIXTHO1ibxUVqxqf2uvMSyG9Or42q3JwA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaINswsLeQ3tWntzCNv//YubG9DP9hzVvFpWH/wi0X bBWbE9ERykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZhI9G2G/27HqrV2aPdMPFHD dm7nPbWiIwlebB6z/0z/vkRk2vstitlAFVG73Pj5ljfM6cxTmSuu5R6/WI9R31pDPtzP8ct3n7U cAA== X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-88-ardb@google.com> Subject: [PATCH v4 25/61] arm64: head: move dynamic shadow call stack patching into early C runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071817_366136_789BB4E3 X-CRM114-Status: GOOD ( 19.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Once we update the early kernel mapping code to only map the kernel once with the right permissions, we can no longer perform code patching via this mapping. So move this code to an earlier stage of the boot, right after applying the relocations. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/scs.h | 4 +-- arch/arm64/kernel/Makefile | 2 -- arch/arm64/kernel/head.S | 8 +++--- arch/arm64/kernel/module.c | 2 +- arch/arm64/kernel/pi/Makefile | 10 +++++--- arch/arm64/kernel/{ => pi}/patch-scs.c | 26 ++++++++++---------- 6 files changed, 27 insertions(+), 25 deletions(-) diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index 3fdae5fe3142..eca2ba5a6276 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -72,8 +72,8 @@ static inline void dynamic_scs_init(void) static inline void dynamic_scs_init(void) {} #endif -int scs_patch(const u8 eh_frame[], int size); -asmlinkage void scs_patch_vmlinux(void); +int __pi_scs_patch(const u8 eh_frame[], int size); +asmlinkage void __pi_scs_patch_vmlinux(void); #endif /* __ASSEMBLY __ */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index dc85dc2ee4ed..14b4a179bad3 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -71,8 +71,6 @@ obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o -obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o -CFLAGS_patch-scs.o += -mbranch-protection=none # Force dependency (vdso*-wrap.S includes vdso.so through incbin) $(obj)/vdso-wrap.o: $(obj)/vdso/vdso.so diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 865ecc1f8255..b320702032a7 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -490,9 +490,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init -#endif -#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS - bl scs_patch_vmlinux #endif mov x0, x20 bl finalise_el2 // Prefer VHE if possible @@ -794,6 +791,11 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RELOCATABLE mov x0, x23 bl __pi_relocate_kernel +#endif +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS + ldr x0, =__eh_frame_start + ldr x1, =__eh_frame_end + bl __pi_scs_patch_vmlinux #endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index dd851297596e..47e0be610bb6 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -595,7 +595,7 @@ int module_finalize(const Elf_Ehdr *hdr, if (scs_is_dynamic()) { s = find_section(hdr, sechdrs, ".init.eh_frame"); if (s) - scs_patch((void *)s->sh_addr, s->sh_size); + __pi_scs_patch((void *)s->sh_addr, s->sh_size); } return module_init_ftrace_plt(hdr, sechdrs, me); diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 7f6dfce893c3..a8b302245f15 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,7 +38,9 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := idreg-override.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o -obj-$(CONFIG_RELOCATABLE) += relocate.pi.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o -extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) +obj-y := idreg-override.pi.o \ + lib-fdt.pi.o lib-fdt_ro.pi.o +obj-$(CONFIG_RELOCATABLE) += relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o +obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/patch-scs.c b/arch/arm64/kernel/pi/patch-scs.c similarity index 91% rename from arch/arm64/kernel/patch-scs.c rename to arch/arm64/kernel/pi/patch-scs.c index a1fe4b4ff591..c65ef40d1e6b 100644 --- a/arch/arm64/kernel/patch-scs.c +++ b/arch/arm64/kernel/pi/patch-scs.c @@ -4,14 +4,11 @@ * Author: Ard Biesheuvel */ -#include #include #include #include -#include #include -#include #include // @@ -81,7 +78,11 @@ static void __always_inline scs_patch_loc(u64 loc) */ return; } - dcache_clean_pou(loc, loc + sizeof(u32)); + if (IS_ENABLED(CONFIG_ARM64_WORKAROUND_CLEAN_CACHE)) + asm("dc civac, %0" :: "r"(loc)); + else + asm(ALTERNATIVE("dc cvau, %0", "nop", ARM64_HAS_CACHE_IDC) + :: "r"(loc)); } /* @@ -128,10 +129,10 @@ struct eh_frame { }; }; -static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, - bool fde_has_augmentation_data, - int code_alignment_factor, - bool dry_run) +static int scs_handle_fde_frame(const struct eh_frame *frame, + bool fde_has_augmentation_data, + int code_alignment_factor, + bool dry_run) { int size = frame->size - offsetof(struct eh_frame, opcodes) + 4; u64 loc = (u64)offset_to_ptr(&frame->initial_loc); @@ -198,14 +199,13 @@ static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, break; default: - pr_err("unhandled opcode: %02x in FDE frame %lx\n", opcode[-1], (uintptr_t)frame); return -ENOEXEC; } } return 0; } -int noinstr scs_patch(const u8 eh_frame[], int size) +int scs_patch(const u8 eh_frame[], int size) { const u8 *p = eh_frame; @@ -251,12 +251,12 @@ int noinstr scs_patch(const u8 eh_frame[], int size) return 0; } -asmlinkage void __init scs_patch_vmlinux(void) +asmlinkage void __init scs_patch_vmlinux(const u8 start[], const u8 end[]) { if (!should_patch_pac_into_scs()) return; - WARN_ON(scs_patch(__eh_frame_start, __eh_frame_end - __eh_frame_start)); - icache_inval_all_pou(); + scs_patch(start, end - start); + asm("ic ialluis"); isb(); } From patchwork Tue Sep 12 14:16:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381789 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A1921CA0EEB for ; Tue, 12 Sep 2023 14:19:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=f1NWoL1OCSDO4XcCmwmia+thL5Y2sCS9NbSVOOnAlJw=; b=FLx8wbh4FrxztM1Kxg43YDfQaO VO7eKIihgKmur7hdDmrily76EDrCNvL4AECdXHLKsvlnjn3RoQEJAYjiyWy38Zn/aVZ+Ht7+Njsoy B84VKQzFXQZTIkXPhjLSYr/XzyMgP1HXbDA9AHtx9ozDMz7ZorhDgYfum05DGiPdxsxSALNc7CUze m+GMv5jQ1RJdC9qaQzzfkxXU8qWNSP1A3TYdhkfyhaOjwia5SoqwA68GXnPvU2m3A2CtmBv4S9OZg GueXayzy6/81bCJMGnpCKwUuS9XMEZoKNO2abLQ89RFZZLq/2fLP5C1JWsTMkHr7LVCv/QwH+lPAw aOzkF/4Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4EH-003X8N-2Y; Tue, 12 Sep 2023 14:19:01 +0000 Received: from mail-ed1-x54a.google.com ([2a00:1450:4864:20::54a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Dd-003WXh-0X for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:32 +0000 Received: by mail-ed1-x54a.google.com with SMTP id 4fb4d7f45d1cf-52c03bb5327so3791658a12.0 for ; Tue, 12 Sep 2023 07:18:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528298; x=1695133098; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xEzGWOTV0DR0iUim1KYbpl164MrytvFFLZfT6M6cUqI=; b=dVft35NCYF8F1VNRIwc5bGg3VYdRrgDWT7BMcRL4LL6sIMM5pNsLVILr1l/BCDtYa0 vQhsKB4372zPkbPBmTqLJzCwkIIzXR1ZzD/+8KN7QSZh4ki1qAb4zfXJtoMvibdiNXbE 2DeKzG+8WDJBz/jPCTvnys24ItlTNB+F+K62rkJFmcdmVSgars+5hjk2WAaR3Wr1G2Wc akJiZohkG1NknIduOpwVGzj+bQyuI01s92TiSbTpo/m8E5HmXzt03zUT94oCLsuPPk1O FXEYLOx5d2xPzcO2MPOlvxDM/8Xriw1VsbbYgKpaWONCMD6euzn8v59THQvhqDHZSnj9 VeOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528298; x=1695133098; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xEzGWOTV0DR0iUim1KYbpl164MrytvFFLZfT6M6cUqI=; b=e+CUA7axkURNAOEPhN44bZUfFT4cam9YYnqZce1Ry/UbF9/nUvtSB76dL+fKt5EQiL hXKtdzNzWDEUSxvIa0E+HSgKbcunVlm/pf137UjuiaFm4mszG3XSIOPGc6l1jfxdSC09 YDcFi1YTOrNc73ltFiR8Pr8ufc0ADCSZlVSOh4tuGs/66qykbxockhp9JR8/TugURaPS tPFHuzVs0LlZVkFjhdm99Sib432onHXlOkNvqjJvC+IcRtxNRjKfHo77kSOiVnXv7BLP qxu3zjf60N+W8R/Dq9BJFq42qMJ11SLUbHwmFKLt2tEPm9C8WZ3fxm37ZqoKg5r+5gS9 /M3g== X-Gm-Message-State: AOJu0Ywsoy81VbtYICaD+mVWqCfOPJ8aWOGY74c+/7Bpv8qc/aFyGVmJ tL8xP+jvTVYgBcq2RB14RqmavUaAmAytCQe2j0+YegaocbkyZSJA9YUgnle7JkYs/jDeGJOwsCh 8VP9IPhdljK3qmpw1e73o3BeDch6rZ0waNCrAlWfVzRW3ywnp0tU6Y8j2NsLBRB8yDv3ivBL4hD k= X-Google-Smtp-Source: AGHT+IGWnU8u0B1jr1sDlL5ZC5ur4Yk8xQJqdHJjOWFbvyHpFfkggHQAcUqIlgqtAzt0I50/L6ih85nI X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a17:907:1587:b0:9a1:b85a:1a05 with SMTP id cf7-20020a170907158700b009a1b85a1a05mr96317ejc.12.1694528297912; Tue, 12 Sep 2023 07:18:17 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:16 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3522; i=ardb@kernel.org; h=from:subject; bh=WpRIXhqE8P+ptTCMt5CXywczxDNmLCqMW7ThFtxeIPs=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaOO5zyXz5pXO4xD52/685R2bb6VszYfuCuk3cXm7b rw6OkWuo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEwkO4+RYeOUlWfLTDhkE8Mz gtK7bvxsW7JXWcK3/3R47kaZq3qeYQz/FPa1XuXxsqqYlba08cD2x3+PWgbenv02ZpOaStp7thh TfgA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-89-ardb@google.com> Subject: [PATCH v4 26/61] arm64: kaslr: Use feature override instead of parsing the cmdline again From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071821_244939_DF15A5F0 X-CRM114-Status: GOOD ( 18.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The early kaslr code open codes the detection of 'nokaslr' on the kernel command line, and this is no longer necessary now that the feature detection code, which also looks for the same string, executes before this code. Note that the pseudo-feature's mask can be disregarded: it is used for true CPU features to mask the CPU feature register, not the value of the override. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 8 +++ arch/arm64/kernel/kaslr.c | 4 +- arch/arm64/kernel/pi/kaslr_early.c | 53 +------------------- 3 files changed, 10 insertions(+), 55 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 96e50227f940..6c0e2cb710d0 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -921,6 +921,14 @@ extern struct arm64_ftr_override id_aa64isar2_override; extern struct arm64_ftr_override arm64_sw_feature_override; +static inline bool kaslr_disabled_cmdline(void) +{ + if (cpuid_feature_extract_unsigned_field(arm64_sw_feature_override.val, + ARM64_SW_FEATURE_OVERRIDE_NOKASLR)) + return true; + return false; +} + u32 get_kvm_ipa_limit(void); void dump_cpu_features(void); diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index 94a269cd1f07..efbeb8356769 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -16,9 +16,7 @@ bool __ro_after_init __kaslr_is_enabled = false; void __init kaslr_init(void) { - if (cpuid_feature_extract_unsigned_field(arm64_sw_feature_override.val & - arm64_sw_feature_override.mask, - ARM64_SW_FEATURE_OVERRIDE_NOKASLR)) { + if (kaslr_disabled_cmdline()) { pr_info("KASLR disabled on command line\n"); return; } diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index 167081b30a15..f2305e276ec3 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -16,57 +16,6 @@ #include #include -/* taken from lib/string.c */ -static char *__init __strstr(const char *s1, const char *s2) -{ - size_t l1, l2; - - l2 = strlen(s2); - if (!l2) - return (char *)s1; - l1 = strlen(s1); - while (l1 >= l2) { - l1--; - if (!memcmp(s1, s2, l2)) - return (char *)s1; - s1++; - } - return NULL; -} -static bool __init cmdline_contains_nokaslr(const u8 *cmdline) -{ - const u8 *str; - - str = __strstr(cmdline, "nokaslr"); - return str == cmdline || (str > cmdline && *(str - 1) == ' '); -} - -static bool __init is_kaslr_disabled_cmdline(void *fdt) -{ - if (!IS_ENABLED(CONFIG_CMDLINE_FORCE)) { - int node; - const u8 *prop; - - node = fdt_path_offset(fdt, "/chosen"); - if (node < 0) - goto out; - - prop = fdt_getprop(fdt, node, "bootargs", NULL); - if (!prop) - goto out; - - if (cmdline_contains_nokaslr(prop)) - return true; - - if (IS_ENABLED(CONFIG_CMDLINE_EXTEND)) - goto out; - - return false; - } -out: - return cmdline_contains_nokaslr(CONFIG_CMDLINE); -} - static u64 __init get_kaslr_seed(void *fdt) { static char const chosen_str[] __initconst = "chosen"; @@ -92,7 +41,7 @@ asmlinkage u64 __init kaslr_early_init(void *fdt) { u64 seed, range; - if (is_kaslr_disabled_cmdline(fdt)) + if (kaslr_disabled_cmdline()) return 0; seed = get_kaslr_seed(fdt); From patchwork Tue Sep 12 14:16:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381788 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4BBB3CA0EEC for ; Tue, 12 Sep 2023 14:19:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=drWD3p6gd3obIzDtlAMfGgPmbCz27KxPQfpaWn724/E=; b=Ac6jrl34hh1eg0226BIvAkIyQm df5ignN0HWn2JFWKKNfFq5Wi7kLt/VGr0w+EXLi3L9DjmyJd+xHb6DP1+fPEqzyQESQY47QjYZw49 /ucxhTzCO5BxzCtiSi12aVW4TPdzwk94CwjcdsloevL3c7F/CtMvSqB0M/fwlcv8AiLEaO+bYx9fs H6BMEJAFeyOFZtIeFEY/e+sf1mvQid6WV/19784/RECBEz60PYrK+7AUrWPzJUXTUaIJoOS5mekRt 3U8HU5Mcg3Vtpj2C0EoTTLBEPhb9980sROr5xEb9DftcVdEWy01Aeog+pBnTMTPkNNUAeNyAiGhm5 XSaLG9Lw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4ED-003X43-2O; Tue, 12 Sep 2023 14:18:57 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Dn-003WhY-1Z for linux-arm-kernel@bombadil.infradead.org; Tue, 12 Sep 2023 14:18:31 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=7FjEszfzmjc49IhSXGtGPOxjDnI19WOUkMh6yagrzHg=; b=GXDunKdjAd67o4+UkTN/U37/HE cilCux6lDF2Ju81dThjzXnIMimqbdvG9YYfvYXgmEJIC5ihqGffIC4R4HvGsqpVlngQyUu6RKpme+ iA+7veqNvDAl0PdrYJxvlt4q4zV0cilP3gx8VF3lQhkLLH1MFSZon5UnmkyvsD+QjR0F3oCruNI+n mx4wl5/by55O/7Xjb2j2/wJx+7MhmEQa4+gd+yVePJITZhMWRQ1sfF0T/9VoLL1Q8foLU54c9SQ57 WWPq0rAzqjBIcs5RlVVsj/nN96CE7ZufU43DwTMf8KLBv5yrWG6uHNiY+pwZDhHizK+ZN518PnncB hwRTjTPw==; Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4De-0069yB-1Z for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:29 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-31400956ce8so3739496f8f.3 for ; Tue, 12 Sep 2023 07:18:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528301; x=1695133101; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7FjEszfzmjc49IhSXGtGPOxjDnI19WOUkMh6yagrzHg=; b=qIKXX4dNw3mWO3JG7NqEYotCtR1A1Pxi+QNP4aV/57h8a3q0WWOkzQ5m/C+yDf7lBr JxeX+p2KhOzMxMIWPTomRHbjyyQyrDWhcrlZjKFgvrHwolVQRXA9cRdaVARrFf0s0oZV bIE0vPecdGbk6mqUdWYAyPeGALP51iTMLJHSBaIR6DtGftvk//I7o3p30RVNyJ1jPc+E Vhz7wWELYeOxqxEF7T8zDk4TmkvKkQwdLEp0vbvo7YCUSunssYf0LFnGhXaxFQlznRuW CKxMCQFRwUQSsIDEfXCvu/gi47jOxMr7bAlFQF5q3wEp+CY0lsigCjP5lGZKugi/EDrT 74zA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528301; x=1695133101; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7FjEszfzmjc49IhSXGtGPOxjDnI19WOUkMh6yagrzHg=; b=f/C7uaKn1a8bJc+o2EXoqweTeiAi46EkSyVwgOr3yjOiE4qrgrqrdwXKGQZkRV0mkf yjNjxGOi7gi6cG4HFlukQeWU1m8oNM61SiDnZlzrj3E/uPbqvxQDVNgbWvRaySfKDrTp TmRZI4YfxGUUIDxMVGL/Xkewc56QJJPo1VaGpJsMgDgpau9GUhcNZykYr7ZD0lJHxM7+ +nx8DZbKj5Gztl/F0waxGYyG/f0nxgJZGzrPkaUOOxQhZx9/94XWRAcdHiPZE8vq1EPi IT1aDJNqzBAeQLmgwo5hdXQl90rcYR6gA0buUgnVLBvlXPEE//nwUQuyHgHvgBXpJyB/ OB3w== X-Gm-Message-State: AOJu0YwcdiY0HV88ngTOYv4HEGkasKq3aSm4d4lD+Ltyh7f+tQLQePkl EAnWnQbXIeVhDeId+2ad2+szQZizS9GObb/5V6fjaTSYBPNLGVFmd860nTAVJggCKlpS2nSO0so JcQNBVICTtYedJ++cinDuNGm2F3Fa9Ru2YouXEuZnDKbsgYPfpiPmFbfnPC6sXeO34QKePmnSzZ 0= X-Google-Smtp-Source: AGHT+IES5PWxRLxFG++NigTsupuiwB7aNqkQlV9xhobHaaVpD4feQ1bs8SqxcCDxDc4okiqzfv/QnGnQ X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:4111:0:b0:317:6846:2347 with SMTP id l17-20020a5d4111000000b0031768462347mr145463wrp.7.1694528300805; Tue, 12 Sep 2023 07:18:20 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:17 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1572; i=ardb@kernel.org; h=from:subject; bh=OWBJokslpSrgiYKe3RTxMg3fV08KqjDzi3T5U/y/rqk=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaNOZrXIOMvctfh6awSMjrv1S7E7qtnu5c1efWyrUV fHCYe/XjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjARpkyG/56v2U79qSyTW3T/ dMvElZOXlKzYGSrxbL0P35xzc5+cOqbNyPArfWWT8KyFQT0XT+6XfvnG4kjT7r+RC0/t1Fi1ewO /rxcLAA== X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-90-ardb@google.com> Subject: [PATCH v4 27/61] arm64/kernel: Move 'nokaslr' parsing out of early idreg code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_151824_177872_3374E6A0 X-CRM114-Status: GOOD ( 13.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Parsing and ignoring 'nokaslr' can be done from anywhere, except from the code that runs very early and is therefore built with limitations on the kind of relocations it is permitted to use. So move it to a source file that is part of the ordinary kernel build. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/kaslr.c | 7 +++++++ arch/arm64/kernel/pi/idreg-override.c | 7 ------- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index efbeb8356769..1da3e25f9d9e 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -34,3 +34,10 @@ void __init kaslr_init(void) pr_info("KASLR enabled\n"); __kaslr_is_enabled = true; } + +static int __init parse_nokaslr(char *unused) +{ + /* nokaslr param handling is done by early cpufeature code */ + return 0; +} +early_param("nokaslr", parse_nokaslr); diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index bcba0ce71af0..26961e0f94b7 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -198,13 +198,6 @@ static const struct { { "nokaslr", "arm64_sw.nokaslr=1" }, }; -static int __init parse_nokaslr(char *unused) -{ - /* nokaslr param handling is done by early cpufeature code */ - return 0; -} -early_param("nokaslr", parse_nokaslr); - static int __init parse_hexdigit(const char *p, u64 *v) { // skip "0x" if it comes next From patchwork Tue Sep 12 14:16:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381790 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0BA2DCA0EEB for ; Tue, 12 Sep 2023 14:19:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=5rfyGfJQ4XMClbuc6u/xhg+YfYH78Snc95vAurAaEZ8=; b=brBp+txZn2GsKo93NzVDqq/k5k EfgvE+o47NUqTb09Z9mzrYxRCW/8QeI8Q0xe37WLjpYGuyQwYaE6aQMLRji5riuVrqY+xE8jdiYx4 jMOAdn+wpHZNK4jpuq2KqJTHSVrZJVPyxTCS+5dMoPkVsAExTVHyGRFfRjYjn1JACplF3/z5Uy3ob 1ZDoM8EHd0/Enot9R4oHC+Vmd3Gn94muuSBllj6zkiKF0sEL74/P4UginVMpwLLH0p5mIaES5+Qyd GYXu8r0DaTVQVGOVXZ+FyN+oq+6RDkt3qRubeQq8Alf+biaHsM9Qz1CPuMz6WLE3tH+/qaXDjPyea Kqw0DQLg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4EP-003XDg-0H; Tue, 12 Sep 2023 14:19:09 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Di-003WcY-1k for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:42 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-31adc3ca07aso3770136f8f.2 for ; Tue, 12 Sep 2023 07:18:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528304; x=1695133104; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7s7pai1xoVt13aBh9Ezg4YG1E//1US9GWQnzYNAW+dM=; b=TVV0+99OG21l2KRdPjccN1SZhips+fbML+O5YMB7aEtvju5MrC9sJhfZtvll+AVyGg s6jlb3ZtOVk2dG6pOSfc5aFsBm6JSc5eYjViAva4wvsX0LzEMoanbqH/aZ85oDQQ6LWl PKYKub370cFNp1ZQ4ZAIbEfc3fjb152NFwJF6KPapESnJPTgsnXY6Tj+ZfqY1v6beINr LWHcMi9U655mYOI5B20ABaIUpVO3WLhqOuG5omKUR2Q0dEXYWTRxNLrPWZ+X/Fg8GkrJ YG4GafNv6KAWPB0VJ41C2vVBmddWa3+kwhpTvNH2x5Lmg8ZrtTZX9EvF3kmpkvMISx5w Ep9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528304; x=1695133104; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7s7pai1xoVt13aBh9Ezg4YG1E//1US9GWQnzYNAW+dM=; b=FO94C8v+CVTkUTerhPsll+kUqC+O281PcIVD5PfzP1rQTbOMhWCFdFrfwxI6Ydnjvf 27suXmVJ/t5fSywCczMjtpBdxVb5k8P4l/P0PytdzqymLLF145usKUrW78b95/fuE+Yi V6qmnLuuxC3uw06FkZCvpSp1ULm4DAEwGLfVb0A1ZbMslknM6kH6qKmZOFNLdlL+M/5p phU4tAKGmhSCe8At0vqUv/NBvIgUO49KCZRXUrlzUJOOTucT/9mN3ns6V1z4tinLODLt T+5VWwQp13/AMHbamca7M6dOTmVFBQV9wnJzzO7t3und6OcwbiNfEHuDzUyFgz0dRXzQ 32UQ== X-Gm-Message-State: AOJu0YyZUU7WytrJ/8gCX9dac4YRXe1srDlFhs6E4g4iSRhcXVopFVJI iX98xQxiAex5RVEvVZEdS26YDntGiHjopiQJkx5x8M/ZfykMrNV7798rCgx9yeFgiIwopam87K1 exAhPgs7kCvc2c8AGUf1m2EhyVWEZ8O3Br4wEBkeiyCou2HU6+C8x3K5BoLW1xkbbIFD/KDanY5 Q= X-Google-Smtp-Source: AGHT+IHX6YS+oBegRZpG8nhOj/EiX4VDWma9+zjm6V5x43qT0laV9Z/nmIz9XuxECRjn4pkG66rCC6K2 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:1005:b0:319:8430:f801 with SMTP id a5-20020a056000100500b003198430f801mr146297wrx.2.1694528303902; Tue, 12 Sep 2023 07:18:23 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:18 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1646; i=ardb@kernel.org; h=from:subject; bh=mQ9WTXexSe7RDVaszyU1z28xUH4JLiVjQ9a8QrnwVNk=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaLP0r1Sl8Og/Wxb49y2aKcp3/+4zkUszq7ecT3E5v nP7CcG6jlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRK6wM/8uafjGae0knuPxl XLrt0+k+nQe3XjyTWH2t3J/55o5Z064y/JVs1Kh0LJ60w+WXwe7bYQ1W3IdUuue05xvt6GQobQw 8ygwA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-91-ardb@google.com> Subject: [PATCH v4 28/61] arm64: idreg-override: Create a pseudo feature for rodata=off From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071826_627197_69BEB42D X-CRM114-Status: GOOD ( 14.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add rodata=off to the set of kernel command line options that is parsed early using the CPU feature override detection code, so we can easily refer to it when creating the kernel mapping. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 1 + arch/arm64/kernel/pi/idreg-override.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 6c0e2cb710d0..54f3dbeb9e80 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -17,6 +17,7 @@ #define ARM64_SW_FEATURE_OVERRIDE_NOKASLR 0 #define ARM64_SW_FEATURE_OVERRIDE_HVHE 4 +#define ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF 8 #ifndef __ASSEMBLY__ diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index 26961e0f94b7..1aa59c01ab33 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -163,6 +163,7 @@ static const struct ftr_set_desc sw_features __prel64_initconst = { .fields = { FIELD("nokaslr", ARM64_SW_FEATURE_OVERRIDE_NOKASLR, NULL), FIELD("hvhe", ARM64_SW_FEATURE_OVERRIDE_HVHE, hvhe_filter), + FIELD("rodataoff", ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF, NULL), {} }, }; @@ -196,6 +197,7 @@ static const struct { { "arm64.nomops", "id_aa64isar2.mops=0" }, { "arm64.nomte", "id_aa64pfr1.mte=0" }, { "nokaslr", "arm64_sw.nokaslr=1" }, + { "rodata=off", "arm64_sw.rodataoff=1" }, }; static int __init parse_hexdigit(const char *p, u64 *v) From patchwork Tue Sep 12 14:16:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381862 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8182FCA0EEB for ; Tue, 12 Sep 2023 15:30:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=1V/k7qC5Ed7nT7hLlec71I66DLEOqoI3vkmo5HDnOQo=; b=1NAMn7AUMTaf1NgW4qYLdrYBTd qNZLQk0F/M04kVAoNJehnSU9tIJwNtS603nhf2k37jjmXWEPf8u0tBm9V2muAmMVp7ahYofsPF/Rd BeS9ZMd6eSwV9+I5RbDiYxDwb9TS1Hr7SfvBE0Q5IHPCZNs25JQgzWz4wBjGF6mqfSiG+9pD3Qbk2 fhMJcMqJfvYdns5JYfffXzzEVTD8Go8ioiFhxXkY/xFX4jTLB1CSHcs+nM27BjOC3LegQCnm5wVK7 pOJNO1fNpVtVWRwvXoEdJwFxRHBFxnbcr/JU9wDqrOmhoRVYM+1Ogjn2JFBJR5AiKmVEJMQjVJYVl cHWkT/SA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg5LQ-003igl-0q; Tue, 12 Sep 2023 15:30:28 +0000 Received: from mail-wr1-f73.google.com ([209.85.221.73]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Dk-003Wed-2f for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:44 +0000 Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-313c930ee0eso3512506f8f.0 for ; Tue, 12 Sep 2023 07:18:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528306; x=1695133106; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LspRidsa25foHXSIlXGmghHoyWSkGb/bgHPv8hJ2YJk=; b=eDOCHyciiuJ/hy9TArOjFPx6R4S/hPhdIkxlc7KXSRMu6uWAk4qAzQABwGQ/44iHQN 4hh56uiiOyu6r2TaNdAAPVkHZb73mTx7ZV7/+26v63sjXyTLwkInrdauoRgwrRjAoLOD Bsan1ROADWwgfJacpqKhk4j89US5avhPxHKOa+4EULckIGYig8ju2lGGP3uUxlSBfWCP VrPiRMP+ZGEEj++DAz8qM0iEuUT+mc7bVeJ/XttjnHloHmiCtGoiuwrjC6r2VebqYBEa bVUmYyk6bF7K5p5R7eQRVwPnWwBByh7oLieGB0lcPQKD0D5ByfCzxL9JinSB2YVUbIYb dADA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528306; x=1695133106; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LspRidsa25foHXSIlXGmghHoyWSkGb/bgHPv8hJ2YJk=; b=IhpVEstgSaeFjMRLTWLdbNOQC1Xi9odEGNsPVm9DgQTikZhKJEom/M0P2EamVBK8/9 Y2y/Fb+FdxDwCjqPRn+YgiPNR5SMA1sM8hP+M+kY7Wca7Gc0/t+65HsInspDd4xFgWpT 33badOX9Tn0t17izF1rHPBWgP6lIaO0pp/bChMeOPwi4JGmHZ4JRc9ocsr9ywQo8xNrP 3DbxREdWhes91voKty5YVc9wjnm339+TnTd547ZiSc9uLHyVOcxWXrB2a1uUcBm0LVRN XFtgaDYGWve0168d9rm88meOz+6U+6O74Mf3lPrJQ+Gf8CdeeKrDBjVUp6EFrQLgv806 DJuQ== X-Gm-Message-State: AOJu0Yx4hp26ghTRvUxotfMnjpf/AwDki5KFMtaPKAA/MDo5CeoHo009 JBDgKyUaHCBjxdVUSjmCwnNtbG6Fy3qf6N6/thy1XhyYWdL8iRCyrApQI67CK4z0s/diSKXI0uE VjUD9g9Xjoo21+20nhSiyHnpYbb3A7zR/vx3TxBtk427Yja1i5KGfeggEncdXJMoFCELFDvoUEH M= X-Google-Smtp-Source: AGHT+IFPYnkhfDmC2DkIe65/AGU9Vi6pxuvU8YSJRbylF172J4nPJqV2+FI32L3kMcYPhe3rUqcuyN2l X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:156b:b0:31f:a664:f871 with SMTP id 11-20020a056000156b00b0031fa664f871mr85968wrz.9.1694528306217; Tue, 12 Sep 2023 07:18:26 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:19 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1935; i=ardb@kernel.org; h=from:subject; bh=rdu64O8XyYBq4cS+NuOmtgmdfARx1xl3BjG7WKGVVsI=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaMvEWYqqtvKt7urdLxgWKPOUnHpT8vj70S/HuCOq/ 3g0KrztKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABNZIMnIcE1F88WPdwYNDyzm MAjaG/lksxg0J1k+Xdjzla/xm8ydWQz/Ayaqc3x58VqiKF/uVmz2qr5vemWZnoeK9i7zva7m48j DBwA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-92-ardb@google.com> Subject: [PATCH v4 29/61] arm64: Add helpers to probe local CPU for PAC and BTI support From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071828_903865_3B00098D X-CRM114-Status: GOOD ( 13.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add some helpers that will be used by the early kernel mapping code to check feature support on the local CPU. This permits the early kernel mapping to be created with the right attributes, removing the need for tearing it down and recreating it. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 44 ++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 54f3dbeb9e80..128861e5d32d 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -933,6 +933,50 @@ static inline bool kaslr_disabled_cmdline(void) u32 get_kvm_ipa_limit(void); void dump_cpu_features(void); +static inline bool cpu_has_bti(void) +{ + u64 pfr1; + + if (!IS_ENABLED(CONFIG_ARM64_BTI)) + return false; + + pfr1 = read_cpuid(ID_AA64PFR1_EL1); + pfr1 &= ~id_aa64pfr1_override.mask; + pfr1 |= id_aa64pfr1_override.val; + + return cpuid_feature_extract_unsigned_field(pfr1, + ID_AA64PFR1_EL1_BT_SHIFT); +} + +static inline bool cpu_has_pac(void) +{ + u64 isar1, isar2; + u8 feat; + + if (!IS_ENABLED(CONFIG_ARM64_PTR_AUTH)) + return false; + + isar1 = read_cpuid(ID_AA64ISAR1_EL1); + isar1 &= ~id_aa64isar1_override.mask; + isar1 |= id_aa64isar1_override.val; + feat = cpuid_feature_extract_unsigned_field(isar1, + ID_AA64ISAR1_EL1_APA_SHIFT); + if (feat) + return true; + + feat = cpuid_feature_extract_unsigned_field(isar1, + ID_AA64ISAR1_EL1_API_SHIFT); + if (feat) + return true; + + isar2 = read_sysreg_s(SYS_ID_AA64ISAR2_EL1); + isar2 &= ~id_aa64isar2_override.mask; + isar2 |= id_aa64isar2_override.val; + feat = cpuid_feature_extract_unsigned_field(isar2, + ID_AA64ISAR2_EL1_APA3_SHIFT); + return feat; +} + #endif /* __ASSEMBLY__ */ #endif From patchwork Tue Sep 12 14:16:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381792 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 55C9ACA0EEC for ; Tue, 12 Sep 2023 14:19:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=b5BAEza6EPIN5mecaPNoDTC+ZuJQMEOaHN5B+wEERXQ=; b=pEoDVQ7coSBtdeufHH5JD0WvK7 uU/PpseJaKeLY528cieMY1Hros2nvIfduBNbNxWfEkFwHhu6J6e2C2sfjabql2qsiKEzE4IvAaV0w ifNEvjuGWe1NCz0DqjI9pIMD7XJYogPUp1QqOnzsx4LjleUXS8X959EPzycBcLx2VDm507e2zPrx0 Awo7RfNS1RDxmM4q6GQfrdEArBQsL+LSDrLmuy080NM6DvtNUJsCOOY6NT+bn2kKJ/t7wLWXnzGKX Bqef4DWbI+T50nbRZB4J8R9ecNZFggmVWC2wGgzs2waZ7jZUBxzlPYzx/OhlL9YQpZjKXamg5zblD LdPcF2ug==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Eb-003XOU-0u; Tue, 12 Sep 2023 14:19:21 +0000 Received: from mail-yb1-f202.google.com ([209.85.219.202]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Dn-003Wgb-0N for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:45 +0000 Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-d80256afb63so8011404276.0 for ; Tue, 12 Sep 2023 07:18:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528308; x=1695133108; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/uq8kNxUep67TUPf7UsnzPOBM66r60ifgQ29Br+vAno=; b=P8ljh2aEQw/kWreww78zyXPV7UqXI9Aq5qnSdAnaqXIjGUht7FGGzPO7OGYpAjUUs7 ia4Io+i0D8c++qUrmAQu9/39lpct5ZIk86Qk2wjiATQyo2GOb7RKXdFXsQDMN3RzlneH Im/dZrSLfIoPuUZAXJ6JMw9gED6aX6g8Y3BeqBputXEI4lGtzhaMsByTI1JwaphFDswI cleOTxpdqCXtmiHGr55ujk48EQzHR9c4fiZhSA+OdJsrmZwejzPNbSaYGObDdUljx1gI nuq/xaOtAxvyj0WBTFEaQJH+soT/5FJFOzDkNuhwJmJIwf96CLq3aQEPdkQWLG5dOUCF yxXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528308; x=1695133108; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/uq8kNxUep67TUPf7UsnzPOBM66r60ifgQ29Br+vAno=; b=Rr/+Inf4/jcDhKbbD3HdyN+6vXk4hx9BeSTM/MOeDIZsCPTXb8zBfB7MzhP4iXCb4r OPWQEbYZGFI9ehVmn+J4lB7mvu4HxB4KD5cCuwhFAaOHOTbVSCscyMdUe7mlAQ7ipyTD lDD+GaI7eqWdf7luBn/8RJJ4BOuSN01UVFdZc6YrF7X0FNrlDD2zwwwaPyE9ffIdNwjq JPYvYCLOy2Qzrni5Lb2P5qc5PufbyvnNcwp4gjq2/Xbb8cp43PPVIICpJOmg5bnAwkih cUKbuygjZYh81gfCB4e0Ytr7DGGlEKEGnvRI2lWTqyw7fKq0KeSnUBp5t2PAnL7JycL3 UQAA== X-Gm-Message-State: AOJu0YzmaMD2bxkJWv6E7jNzJdR+ScC0HS4t+HQZ8swgq8wrBO/lKuMo GLoXe1IhAx8WyNmnpENVK1bieoBpH9KahuXjLdg++CBlj66iz+w/hB+KuxzxxSliSHTj51eMl+1 IOvW78FAHMQVR47NWzkMvUfYfmEXSN86vL+RKopFO2XlHi4v/qpQmqiPg/5hL8ge4sdtdxeR3Oy U= X-Google-Smtp-Source: AGHT+IG3kQ7356/kn+51Tp4kPdRwAhfJLIusASHs0NjrMrtTkpRoi+I02j4JMX9eN6I46OZLVZqQrA8+ X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:77c1:0:b0:d81:1cfe:70bc with SMTP id s184-20020a2577c1000000b00d811cfe70bcmr40647ybc.3.1694528308538; Tue, 12 Sep 2023 07:18:28 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:20 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1834; i=ardb@kernel.org; h=from:subject; bh=CKEH2MCk6aR4/ylPquP0IGzEdUdPYBvTsIrNtoxJujA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaOt5pdm65h1Vl67OfX1vyxPzmZ+9PzazqCyeaBokP SlkIvvejlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCR+Y4M/yNDt669zC69bC6b edPiU+8T260cnkToyL9r1tHYfTmWN5CRYUPzywCT//tXzd6tk1ryfntOr2ZA8IQjJ7ZdWDl5qWH YZz4A X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-93-ardb@google.com> Subject: [PATCH v4 30/61] arm64: head: allocate more pages for the kernel mapping From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071831_241737_C8FDC938 X-CRM114-Status: GOOD ( 14.81 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In preparation for switching to an early kernel mapping routine that maps each segment according to its precise boundaries, and with the correct attributes, let's allocate some extra pages for page tables for the 4k page size configuration. This is necessary because the start and end of each segment may not be aligned to the block size, and so we'll need an extra page table at each segment boundary. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 83ddb14b95a5..0631604995ee 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -68,7 +68,7 @@ + EARLY_PGDS((vstart), (vend), add) /* each PGDIR needs a next level page table */ \ + EARLY_PUDS((vstart), (vend), add) /* each PUD needs a next level page table */ \ + EARLY_PMDS((vstart), (vend), add)) /* each PMD needs a next level page table */ -#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE)) +#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE) + EARLY_SEGMENT_EXTRA_PAGES)) /* the initial ID map may need two extra pages if it needs to be extended */ #if VA_BITS < 48 @@ -89,6 +89,15 @@ #define SWAPPER_TABLE_SHIFT PMD_SHIFT #endif +/* The number of segments in the kernel image (text, rodata, inittext, initdata, data+bss) */ +#define KERNEL_SEGMENT_COUNT 5 + +#if SWAPPER_BLOCK_SIZE > SEGMENT_ALIGN +#define EARLY_SEGMENT_EXTRA_PAGES (KERNEL_SEGMENT_COUNT + 1) +#else +#define EARLY_SEGMENT_EXTRA_PAGES 0 +#endif + /* * Initial memory map attributes. */ From patchwork Tue Sep 12 14:16:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381794 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A7D8CCA0EEC for ; Tue, 12 Sep 2023 14:20:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ri/qulmyjKTTSnvj5u8HqJTJs1Y/671FIk1dW1I/MUQ=; b=Nvd5rlZYXzCQAxrpj/hWMdYu7j CBUNbdFW+gJFTp0dNt/OHtYUp6xVz60ZhW5ksbG3bFf3C5EsbjQwmHLYiC9jVun4oLljkUh7eOjar TBxQ8WBsgL61tu9TZ7lK5kkf9UC80llb7JHbBwS/NdDGqCH3iXDTOm+NSf23IbpFsJL6cz5o0QD3V z5s2Cd+E0np2iFp7CWNU3TpPjCKMKTOs0a6vEtCuOFBhXxWz4IEupiueMgcpYwfuZKrADmb8HRz4H 14sbgpZ4QePG9mGAabHioiLs8eqMnRyNmvgl/EmhvJTm22dQ3BQ3LIlX2C+LhD1s0Ma1iKwDMxKg2 lDfHb0rw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Ef-003XSJ-0S; Tue, 12 Sep 2023 14:19:25 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Do-003Wih-2e for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:46 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-597f461adc5so62612107b3.1 for ; Tue, 12 Sep 2023 07:18:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528311; x=1695133111; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WTQFaUZLVUqp+Nb8PeZm3J4OERkW9kq8/gs7s5p0Y2g=; b=hhJG2wfR57htYQ181XsL4J9OQWex4Iy6C/65ELeSvzye4ntk8Bo4Y0o8wW7Mf3O7kn 0V4KvWFbzz9lsQDLu2G19kNuLgofATNkdokz/cWMWxdBTU2Lc53G1vDeqWYsrF9fK6Lb qOER8Fl1RF5ijTUfmUNPT2uMZb7LfphfTYVRTDqlXJ6SSGJxDmy2tZWrbTrZgD3MNz5z OZ60MZ8WU9ygFLMkOp89MtKkY6E6JuyqXNHkdarA2d35yh66oQyOMTHcvdk+GKykzRgz pPtUYwtD/JS9Niib5+WcV3ooimZUl08/bcfhAK+EGQJauN/rZrfjjlcBgplyCZPda6M2 t+Yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528311; x=1695133111; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WTQFaUZLVUqp+Nb8PeZm3J4OERkW9kq8/gs7s5p0Y2g=; b=p9aPPkz2pxFCun68OHjotIp35JcE+wk3GHWxI1okQ8wBKSpwBy7OVqtPq2ky07gCpV MDFeC8qCp7LX+efzFm8LhEl5Gf6dfy78N9nTx1C4OhPJrOmkOhgXQ0TvEtZ5b8/dykWc O/XTiboq7Ejw3ZkcLiP74LY6kvKYocXpUDII9I1IWPVrhyDMVIgE5m/yBG4ePvQHxOam pc62smo3+4lPThQbDjhbb3rKabCsyUwBd+P+VAyR99+H6Nd0CqJES2tq9rYnNILkJWGL Av9T28UuPoj8ZByXkQx4dpG8nru1JT9gLFfovffIXnbapujP8lHYgpXmAz3bt8UkuFBm h1cg== X-Gm-Message-State: AOJu0YyroY4mqSSwMMqqdf7DRt4Y9jmtrhIEowCjB17D9tTMDVWzvJ2x SLdEN9wt3gyiJnuAj29B89gPxLDn4rdYJzX63JY/6VcxQDPT2u3kecwlF4E3LJXiOXQFrSDlxpz TtMaCZGoJkFWVFN1wF6TLQGOJqmRIFWYzfGuQhgWzsBL3v65MG3g6jPfA7xowAubbs/oRgBLj4i A= X-Google-Smtp-Source: AGHT+IHQ+ZdqBGMKUFHgQkn++intjoHO8H6baEprppn4bAe7+sJy+FZL5GD61fMhoMKijBaVVML7lGZd X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:c548:0:b0:56c:f8b7:d4fa with SMTP id o8-20020a81c548000000b0056cf8b7d4famr298363ywj.7.1694528310794; Tue, 12 Sep 2023 07:18:30 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:21 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3177; i=ardb@kernel.org; h=from:subject; bh=3t5eK0yXAKEY06tW4A49CunV4PkgMuSeHKCOWhSWhg0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaNsvA9953D9md5/KCVm1N8JU1OP7U8acKttHs4WvB 90tiM7vKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABM538XIsHXGZqUExUXiDAfv rTU5ymSf+sXctMj+w6EXB6Pyulxu3mJkuPnO7Vr1OpapW2raZ3CJ60zbacllm6bit8002u3kppI wPgA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-94-ardb@google.com> Subject: [PATCH v4 31/61] arm64: head: move memstart_offset_seed handling to C code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071832_897076_57C7A2DD X-CRM114-Status: GOOD ( 14.86 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that we can set BSS variables from the early code running from the ID map, we can set memstart_offset_seed directly from the C code that derives the value instead of passing it back and forth between C and asm code. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 7 ------- arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/kaslr_early.c | 4 ++++ 3 files changed, 5 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index b320702032a7..aa7766dc64d9 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -82,7 +82,6 @@ * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob * x23 __primary_switch() physical misalignment/KASLR offset - * x24 __primary_switch() linear map KASLR seed * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register */ @@ -483,11 +482,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) str x25, [x8] // ... observes the correct value dc civac, x8 // Make visible to booting secondaries #endif - -#ifdef CONFIG_RANDOMIZE_BASE - adrp x5, memstart_offset_seed // Save KASLR linear map seed - strh w24, [x5, :lo12:memstart_offset_seed] -#endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif @@ -779,7 +773,6 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RANDOMIZE_BASE mov x0, x22 bl __pi_kaslr_early_init - and x24, x0, #SZ_2M - 1 // capture memstart offset seed bic x0, x0, #SZ_2M - 1 orr x23, x23, x0 // record kernel offset #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 011a9dd93bd2..f2c06f1a9e70 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -44,6 +44,7 @@ PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); PROVIDE(__pi_id_aa64zfr0_override = id_aa64zfr0_override); PROVIDE(__pi_arm64_sw_feature_override = arm64_sw_feature_override); PROVIDE(__pi__ctype = _ctype); +PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); #ifdef CONFIG_KVM diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index f2305e276ec3..eeecee7ffd6f 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -16,6 +16,8 @@ #include #include +extern u16 memstart_offset_seed; + static u64 __init get_kaslr_seed(void *fdt) { static char const chosen_str[] __initconst = "chosen"; @@ -51,6 +53,8 @@ asmlinkage u64 __init kaslr_early_init(void *fdt) return 0; } + memstart_offset_seed = seed & U16_MAX; + /* * OK, so we are proceeding with KASLR enabled. Calculate a suitable * kernel image offset from the seed. Let's place the kernel in the From patchwork Tue Sep 12 14:16:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381793 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AC01DCA0EEC for ; Tue, 12 Sep 2023 14:19:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=9PhXarDoG3bulz9rsEXPjDBv9kRk4gzjzfOg0CakuMI=; b=fCPst9DFb91c7mkG9kElJj/DnG ZjtfMzZ/yDBHZVSlc3zmVlVpvBFN3WF3YdlIEgamhLigan0ZGDZhAFbQIq5IHPpxlZEgV0zKbJkFY D/56UXckhaIO9D5NEdseNRaFuAy8oAw+fUe/nksWkxtmiTbozSZWY8vBNK6WtjHNrounrJ9A2ncnK wxLmJfRP2pV0TGUd+p6AXynsLtfhy4g9VoHwYLAeneNhEN/f3nR9WA+yPl12EiL1LHTzrngrn8xqq RA7CSJ9DZYnQZ5JT9Lmp1IiZgHraj0wWFlmANgQDSdB8fzAED4/NoMFEoOuHOfRRZnnjyrXEdvwIG XbYKNs9g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Eg-003XTv-2a; Tue, 12 Sep 2023 14:19:26 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Dr-003WkJ-00 for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:48 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-d7e79ec07b4so5569611276.0 for ; Tue, 12 Sep 2023 07:18:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528313; x=1695133113; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=iYrAAXbuRoK7lbpKClqf0Z9KrlvpEUh/9+3vTgpo0TY=; b=QxpBZeDEV1esWvdNE8MYJym7/A5SPrrjBpCjLw+/ga+m40HS/xGM21LndaIsUmhpgn 2ilUyuDMp5UKNweRmAVEGSMdwAGKDB6SZbTiOwWdhpjqOHz+JNyomtpiVp9hh6i85L7t J3kE3ATRLosjTl6yTlrBjxbRslojyqRbpCG5FiFBdvqWKlv5+w5s24EY3JIRWS1wDMNh dMh0i1vE++1mPJb4aiZpBfqrzEI8SY4TiGhjnkBpp2rCzoTkuF9a8jDu4MFLtos89h4I DG4Ct9jyWdLuwc229g0t1z442o+BMiY44wMSDHbj33XUpMIrEANS3KtKXrKLS6nFJC7C uVlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528313; x=1695133113; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iYrAAXbuRoK7lbpKClqf0Z9KrlvpEUh/9+3vTgpo0TY=; b=S+ivQb/vqGJU1SmHb7pkmIXx5iWkIOLr4r48+SFblCNMQuCOilzq1AtcnBh4VNA1/x lpg0TJHNfv5FKFxZFAfnFCwptIUFAOt9raxVtMYumV5pyfLkkMEGDZBKnEexsBlDWF5Z c7IuArjCui6JsWk5d4hK0QP4hx/fjPEDjkiRimD29oYmPrZ7Kr0Zu7JcQoycWcZNjM3E k89L0uW5T4rmGZ33N+TpbpS04Xyf8M7B1eTb6GHPe6zusDkF43A+3AhePj0sJqvJ06/p /0Hs2E5rsP7sidTUCBEOcBzBshgqJdB97ubR8GieI8O0TLPhnabN7NdxxkBEETb6r8wi d8EQ== X-Gm-Message-State: AOJu0YxXeORXWviNNEslLlGR3jtSZPLTgj8dZStaaRiBZ8RGZ1ifW7YX XZ6gr+OxPX5orIv+hycDZmPB9lcsKcWQkmgvltRNa9KOwj6e4QzKyNultFInr021CY7dniRz5Lo EoPUOJYfWlk1Sz1TYdKUgcznOqDKu7aAYYCVmLKTwWHDsyNnftdbWKZjcOQAR0eppLA54cvoMyY w= X-Google-Smtp-Source: AGHT+IFqbuZm7mbI9ikzsLYSVtPEqn9c8elQrZ+LhAno3j9wZ62vinTSFaMk9EPT3kcpgdNGHZ2/eWH3 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:108f:b0:d78:3c2e:b186 with SMTP id v15-20020a056902108f00b00d783c2eb186mr294108ybu.5.1694528313222; Tue, 12 Sep 2023 07:18:33 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:22 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5075; i=ardb@kernel.org; h=from:subject; bh=sUQNGAibWz3dErcNZKyYQRoQSQUITT2GMpOs/+vgEVY=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaPspHtOr3yN//lojcWPyrd9bPa/zbtLbudOMezqHv kF6RurHjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRUycYGfr6lpw8cXPlUZYv ppv3/PzWca8/4l2V/S2JjtW394VM3nCe4X9QnOCc7skWs/X8eFZoV8++7Tgxw+F17qrANTeuBEV fv84FAA== X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-95-ardb@google.com> Subject: [PATCH v4 32/61] arm64: mm: Make kaslr_requires_kpti() a static inline From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071835_078931_1EC762D1 X-CRM114-Status: GOOD ( 24.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In preparation for moving the first assignment of arm64_use_ng_mappings to an earlier stage in the boot, ensure that kaslr_requires_kpti() is accessible without relying on the core kernel's view on whether or not KASLR is enabled. So make it a static inline, and move the kaslr_enabled() check out of it and into the callers, one of which will disappear in a subsequent patch. Once/when support for the obsolete ThunderX 1 platform is dropped, this check reduces to a E0PD feature check on the local CPU. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/mmu.h | 38 +++++++++++++++++- arch/arm64/kernel/cpufeature.c | 42 +------------------- arch/arm64/kernel/setup.c | 2 +- 3 files changed, 39 insertions(+), 43 deletions(-) diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 94b68850cb9f..be41054a500e 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -71,7 +71,43 @@ extern void create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, pgprot_t prot, bool page_mappings_only); extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot); extern void mark_linear_text_alias_ro(void); -extern bool kaslr_requires_kpti(void); + +/* + * This check is triggered during the early boot before the cpufeature + * is initialised. Checking the status on the local CPU allows the boot + * CPU to detect the need for non-global mappings and thus avoiding a + * pagetable re-write after all the CPUs are booted. This check will be + * anyway run on individual CPUs, allowing us to get the consistent + * state once the SMP CPUs are up and thus make the switch to non-global + * mappings if required. + */ +static inline bool kaslr_requires_kpti(void) +{ + /* + * E0PD does a similar job to KPTI so can be used instead + * where available. + */ + if (IS_ENABLED(CONFIG_ARM64_E0PD)) { + u64 mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + if (cpuid_feature_extract_unsigned_field(mmfr2, + ID_AA64MMFR2_EL1_E0PD_SHIFT)) + return false; + } + + /* + * Systems affected by Cavium erratum 24756 are incompatible + * with KPTI. + */ + if (IS_ENABLED(CONFIG_CAVIUM_ERRATUM_27456)) { + extern const struct midr_range cavium_erratum_27456_cpus[]; + + if (is_midr_in_range_list(read_cpuid_id(), + cavium_erratum_27456_cpus)) + return false; + } + + return true; +} #define INIT_MM_CONTEXT(name) \ .pgd = init_pg_dir, diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index bdd492cea88b..c91f658b175e 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1626,46 +1626,6 @@ has_useable_cnp(const struct arm64_cpu_capabilities *entry, int scope) return has_cpuid_feature(entry, scope); } -/* - * This check is triggered during the early boot before the cpufeature - * is initialised. Checking the status on the local CPU allows the boot - * CPU to detect the need for non-global mappings and thus avoiding a - * pagetable re-write after all the CPUs are booted. This check will be - * anyway run on individual CPUs, allowing us to get the consistent - * state once the SMP CPUs are up and thus make the switch to non-global - * mappings if required. - */ -bool kaslr_requires_kpti(void) -{ - if (!IS_ENABLED(CONFIG_RANDOMIZE_BASE)) - return false; - - /* - * E0PD does a similar job to KPTI so can be used instead - * where available. - */ - if (IS_ENABLED(CONFIG_ARM64_E0PD)) { - u64 mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); - if (cpuid_feature_extract_unsigned_field(mmfr2, - ID_AA64MMFR2_EL1_E0PD_SHIFT)) - return false; - } - - /* - * Systems affected by Cavium erratum 24756 are incompatible - * with KPTI. - */ - if (IS_ENABLED(CONFIG_CAVIUM_ERRATUM_27456)) { - extern const struct midr_range cavium_erratum_27456_cpus[]; - - if (is_midr_in_range_list(read_cpuid_id(), - cavium_erratum_27456_cpus)) - return false; - } - - return kaslr_enabled(); -} - static bool __meltdown_safe = true; static int __kpti_forced; /* 0: not forced, >0: forced on, <0: forced off */ @@ -1718,7 +1678,7 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, } /* Useful for KASLR robustness */ - if (kaslr_requires_kpti()) { + if (kaslr_enabled() && kaslr_requires_kpti()) { if (!__kpti_forced) { str = "KASLR"; __kpti_forced = 1; diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 4b0b3515ee20..c2d6852a4e0c 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -288,7 +288,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) * mappings from the start, avoiding the cost of rewriting * everything later. */ - arm64_use_ng_mappings = kaslr_requires_kpti(); + arm64_use_ng_mappings = kaslr_enabled() && kaslr_requires_kpti(); early_fixmap_init(); early_ioremap_init(); From patchwork Tue Sep 12 14:16:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381791 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A1D78CA0EEC for ; Tue, 12 Sep 2023 14:19:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=xrVYIynmBtpu2DQ8J/p4JOBXaRtcLbo0eUJlcNS614w=; b=TxWAWtljB0V+FJtwkAwYzgmX/B 872dxySmzYtnZFDed4qn+p+4Z7rM4Ymjkpv3FCkRi5UrmcgrWMLbe64nQwA65xFoy99k24/epDSaR NJyMXA9/i0hcHFlxkO4W3vFvxlL4HKPh5DP376GzR5kcqy9CK9LNU6nVUc6lEIo8N1JvDCKMwPwZP 0WPSaWWEoq9JEMdmq6DzlJ+/BNFaEA3Qdk2AWUjDo8FQpndhtROO8jtUlSEn6n2FaZpzr5mi6yqwq m0D+Fhe8MJqH2Mn+T0Bo2/SrfZkUYg0JLwJP0tDk6WW2h61fYkfYgmh41HcT+yUsHo8UcEMYHysHq y2toW05g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4ES-003XGT-2A; Tue, 12 Sep 2023 14:19:12 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Dz-003Wrt-21 for linux-arm-kernel@bombadil.infradead.org; Tue, 12 Sep 2023 14:18:43 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=0tzN5e26cAQKj3ATntk6jMcJoaaHPgH9QhZbfuF6Ekk=; b=TcNUdq3UEdXGF6XxKpikdPGCrP FhC74lkRaH43ixI6nJDzeW2Gjkmn0q7iMoQrRPlalUAN7IqTbzUElCj9dKr5kfu1k9dv/VMv2+Ph4 orPRuOVIkuUXkodBIPvkx1JzN1OdSvGhg9NLOovUTF0Bsnrhd6pY1juj0xOXz/z5E5akMSZAGmB1z ewCdkdhyelXasnFvZy7jEr7IhsWfH7n708xmw2+p8wxDWKk8ghjA+9x3KmTFtiABddS/9+QNrFdMK x0Rxip16iN2dX6gbp8jQsjXDip8wlevMZzKSd14BvWAiCylanz07Y0HmHFg/69K5Dua4Vxp5n0RQt Oe+Fg3Vw==; Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Dt-0069z1-1E for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:42 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-31f8999d44bso2455805f8f.3 for ; Tue, 12 Sep 2023 07:18:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528316; x=1695133116; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0tzN5e26cAQKj3ATntk6jMcJoaaHPgH9QhZbfuF6Ekk=; b=qvSdKYC9YHgZJ9ZfD7eky0hRHTcl55NwDzH1+dbj5dx8jS5aEFdu6yBzcKJ2atBmI8 nw6ROT68ySh3ECeuft7eJdypwZFJI2HtK8PXqWfHy7tBB21CONppCq9WswDadDglziBL PVc+2e2x8Y1dfZGboHYiO+N7C7ePTATkATkG4m6uHtLnt8cXBD/cbbYX6BY9Sx1OO/rb DxeumRR/A6xJ/XWvQlh6TXUw2BP/iequmReF9ackyzXmWkPkhIH0gyW/nW2RDkc9D9t7 3PlnttgFnpqMbosw7wjoh54xVYlBaGSMIVFKhVNfLSzcPMxEHKAUth1JtB4HCEy/4Z7j 66Fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528316; x=1695133116; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0tzN5e26cAQKj3ATntk6jMcJoaaHPgH9QhZbfuF6Ekk=; b=PAIbNavEcJf4sZXn83BCwzE0fbfQsfk51hIjpgqOkuUlmfESJjwgqlbtS7/3YFQiOD P3pHjY9AyYDMN9wbm65jDQDa6VKbQi4edUBOd2Y8Zil808Vamel/G609f8CmmpRgJpW2 Dv5USLcbvxTT+KhxoMfxpNxwc+sVt6kgbK8UwGtUXotC42BVispokojs341iHBxBEDC5 kiAAGwm4SdJ4z6owLtWX4iM6uOruLqVTjKZA/bXEqK11yTpkPEFy3F1BrU7lrUR29Xwz /pN5JwI6wRrKi7YSMj+GZ4HBCTMPxpia2Vh/h4QhrOOLrJMQGCH+LgZNwbsWpHdtakP9 7Fhw== X-Gm-Message-State: AOJu0Yxb8AWQWfKTIyEsgFJwtKwfIbM2KnXdS62wYWdrsZAQL0Ho7NNO XkBo264EUzteQwgGJk9lOXlrTOl42fidG2WXEOfWIJ9SpTu4UoIXd1Y8Jn3WpQY20IzVJi2smoi KH1hMImJtimjgyQpmq9SOAZYZfis+Fu2o6EQuAYEPviVypvW2d2x9gAMVa4bbUu1esY798iJvst A= X-Google-Smtp-Source: AGHT+IGn7rygNod2R7Mn4Wor1FfVDeFWPKpX38YeOVU4VpemIW6RYI4TJnQqNuF+kgzQig4PCQ5f6kRw X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:4252:0:b0:314:419:64f0 with SMTP id s18-20020a5d4252000000b00314041964f0mr170926wrr.9.1694528315830; Tue, 12 Sep 2023 07:18:35 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:23 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=23517; i=ardb@kernel.org; h=from:subject; bh=vVvCGKWG34VVWidM1+Mv+Z5PJXUjHqEZT/6uzy1NgKc=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaIfNeu6tD+W/5pWd8p3e1/PzsnRK2AFR94JU0aj3B R6f3E92lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgIkE8zIy9Hf8X1W1jTtx8W2e m7PnPUnLChP64jr5Fm90py7PriPsnIwMEz2n7LiXXHDO5rOgwIYpModrthV8U+7RZdx4Iktjs9V KfgA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-96-ardb@google.com> Subject: [PATCH v4 33/61] arm64: head: Move early kernel mapping routines into C code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_151838_981932_819EE0DC X-CRM114-Status: GOOD ( 26.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The asm version of the kernel mapping code works fine for creating a coarse grained identity map, but for mapping the kernel down to its exact boundaries with the right attributes, it is not suitable. This is why we create a preliminary RWX kernel mapping first, and then rebuild it from scratch later on. So let's reimplement this in C, in a way that will make it unnecessary to create the kernel page tables yet another time in paging_init(). Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/archrandom.h | 2 - arch/arm64/include/asm/scs.h | 32 +--- arch/arm64/kernel/head.S | 52 +----- arch/arm64/kernel/image-vars.h | 19 +++ arch/arm64/kernel/pi/Makefile | 1 + arch/arm64/kernel/pi/idreg-override.c | 22 ++- arch/arm64/kernel/pi/kaslr_early.c | 12 +- arch/arm64/kernel/pi/map_kernel.c | 165 ++++++++++++++++++++ arch/arm64/kernel/pi/map_range.c | 88 +++++++++++ arch/arm64/kernel/pi/patch-scs.c | 16 +- arch/arm64/kernel/pi/pi.h | 12 ++ arch/arm64/kernel/pi/relocate.c | 2 + arch/arm64/kernel/setup.c | 7 - arch/arm64/kernel/vmlinux.lds.S | 4 +- arch/arm64/mm/proc.S | 1 + 15 files changed, 314 insertions(+), 121 deletions(-) diff --git a/arch/arm64/include/asm/archrandom.h b/arch/arm64/include/asm/archrandom.h index b0abc64f86b0..2f5f3da34782 100644 --- a/arch/arm64/include/asm/archrandom.h +++ b/arch/arm64/include/asm/archrandom.h @@ -129,6 +129,4 @@ static inline bool __init __early_cpu_has_rndr(void) return (ftr >> ID_AA64ISAR0_EL1_RNDR_SHIFT) & 0xf; } -u64 kaslr_early_init(void *fdt); - #endif /* _ASM_ARCHRANDOM_H */ diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index eca2ba5a6276..2e010ea76be2 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -33,37 +33,11 @@ #include #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS -static inline bool should_patch_pac_into_scs(void) -{ - u64 reg; - - /* - * We only enable the shadow call stack dynamically if we are running - * on a system that does not implement PAC or BTI. PAC and SCS provide - * roughly the same level of protection, and BTI relies on the PACIASP - * instructions serving as landing pads, preventing us from patching - * those instructions into something else. - */ - reg = read_sysreg_s(SYS_ID_AA64ISAR1_EL1); - if (SYS_FIELD_GET(ID_AA64ISAR1_EL1, APA, reg) | - SYS_FIELD_GET(ID_AA64ISAR1_EL1, API, reg)) - return false; - - reg = read_sysreg_s(SYS_ID_AA64ISAR2_EL1); - if (SYS_FIELD_GET(ID_AA64ISAR2_EL1, APA3, reg)) - return false; - - if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) { - reg = read_sysreg_s(SYS_ID_AA64PFR1_EL1); - if (reg & (0xf << ID_AA64PFR1_EL1_BT_SHIFT)) - return false; - } - return true; -} - static inline void dynamic_scs_init(void) { - if (should_patch_pac_into_scs()) { + extern bool __pi_dynamic_scs_is_enabled; + + if (__pi_dynamic_scs_is_enabled) { pr_info("Enabling dynamic shadow call stack\n"); static_branch_enable(&dynamic_scs_enabled); } diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index aa7766dc64d9..ffacce7b5a02 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -81,7 +81,6 @@ * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob - * x23 __primary_switch() physical misalignment/KASLR offset * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register */ @@ -408,24 +407,6 @@ SYM_FUNC_START_LOCAL(create_idmap) 0: ret x28 SYM_FUNC_END(create_idmap) -SYM_FUNC_START_LOCAL(create_kernel_mapping) - adrp x0, init_pg_dir - mov_q x5, KIMAGE_VADDR // compile time __va(_text) -#ifdef CONFIG_RELOCATABLE - add x5, x5, x23 // add KASLR displacement -#endif - adrp x6, _end // runtime __pa(_end) - adrp x3, _text // runtime __pa(_text) - sub x6, x6, x3 // _end - _text - add x6, x6, x5 // runtime __va(_end) - mov_q x7, SWAPPER_RW_MMUFLAGS - - map_memory x0, x1, x5, x6, x7, x3, (VA_BITS - PGDIR_SHIFT), x10, x11, x12, x13, x14 - - dsb ishst // sync with page table walker - ret -SYM_FUNC_END(create_kernel_mapping) - /* * Initialize CPU registers with task-specific and cpu-specific context. * @@ -752,44 +733,13 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x2, init_idmap_pg_dir bl __enable_mmu - // Clear BSS - adrp x0, __bss_start - mov x1, xzr - adrp x2, init_pg_end - sub x2, x2, x0 - bl __pi_memset - dsb ishst // Make zero page visible to PTW - adrp x1, early_init_stack mov sp, x1 mov x29, xzr mov x0, x20 // pass the full boot status mov x1, x22 // pass the low FDT mapping - bl __pi_init_feature_override // Parse cpu feature overrides - -#ifdef CONFIG_RELOCATABLE - adrp x23, KERNEL_START - and x23, x23, MIN_KIMG_ALIGN - 1 -#ifdef CONFIG_RANDOMIZE_BASE - mov x0, x22 - bl __pi_kaslr_early_init - bic x0, x0, #SZ_2M - 1 - orr x23, x23, x0 // record kernel offset -#endif -#endif - bl create_kernel_mapping + bl __pi_early_map_kernel // Map and relocate the kernel - adrp x1, init_pg_dir - load_ttbr1 x1, x1, x2 -#ifdef CONFIG_RELOCATABLE - mov x0, x23 - bl __pi_relocate_kernel -#endif -#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS - ldr x0, =__eh_frame_start - ldr x1, =__eh_frame_end - bl __pi_scs_patch_vmlinux -#endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) br x8 diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index f2c06f1a9e70..bb108f4f9b75 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -43,9 +43,28 @@ PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override); PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); PROVIDE(__pi_id_aa64zfr0_override = id_aa64zfr0_override); PROVIDE(__pi_arm64_sw_feature_override = arm64_sw_feature_override); +PROVIDE(__pi_arm64_use_ng_mappings = arm64_use_ng_mappings); +#ifdef CONFIG_CAVIUM_ERRATUM_27456 +PROVIDE(__pi_cavium_erratum_27456_cpus = cavium_erratum_27456_cpus); +#endif PROVIDE(__pi__ctype = _ctype); PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); +PROVIDE(__pi_init_pg_dir = init_pg_dir); +PROVIDE(__pi_init_pg_end = init_pg_end); + +PROVIDE(__pi__text = _text); +PROVIDE(__pi__stext = _stext); +PROVIDE(__pi__etext = _etext); +PROVIDE(__pi___start_rodata = __start_rodata); +PROVIDE(__pi___inittext_begin = __inittext_begin); +PROVIDE(__pi___inittext_end = __inittext_end); +PROVIDE(__pi___initdata_begin = __initdata_begin); +PROVIDE(__pi___initdata_end = __initdata_end); +PROVIDE(__pi__data = _data); +PROVIDE(__pi___bss_start = __bss_start); +PROVIDE(__pi__end = _end); + #ifdef CONFIG_KVM /* diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index a8b302245f15..8c2f80a46b93 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -39,6 +39,7 @@ $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) obj-y := idreg-override.pi.o \ + map_kernel.pi.o map_range.pi.o \ lib-fdt.pi.o lib-fdt_ro.pi.o obj-$(CONFIG_RELOCATABLE) += relocate.pi.o obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index 1aa59c01ab33..5857e4efad59 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -314,37 +314,35 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) } while (1); } -static __init const u8 *get_bootargs_cmdline(const void *fdt) +static __init const u8 *get_bootargs_cmdline(const void *fdt, int node) { + static char const bootargs[] __initconst = "bootargs"; const u8 *prop; - int node; - node = fdt_path_offset(fdt, "/chosen"); if (node < 0) return NULL; - prop = fdt_getprop(fdt, node, "bootargs", NULL); + prop = fdt_getprop(fdt, node, bootargs, NULL); if (!prop) return NULL; return strlen(prop) ? prop : NULL; } -static __init void parse_cmdline(const void *fdt) +static __init void parse_cmdline(const void *fdt, int chosen) { - const u8 *prop = get_bootargs_cmdline(fdt); + static char const cmdline[] __initconst = CONFIG_CMDLINE; + const u8 *prop = get_bootargs_cmdline(fdt, chosen); if (IS_ENABLED(CONFIG_CMDLINE_FORCE) || !prop) - __parse_cmdline(CONFIG_CMDLINE, true); + __parse_cmdline(cmdline, true); if (!IS_ENABLED(CONFIG_CMDLINE_FORCE) && prop) __parse_cmdline(prop, true); } -/* Keep checkers quiet */ -void init_feature_override(u64 boot_status, const void *fdt); - -asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) +void __init init_feature_override(u64 boot_status, const void *fdt, + int chosen) { struct arm64_ftr_override *override; const struct ftr_set_desc *reg; @@ -360,7 +358,7 @@ asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) __boot_status = boot_status; - parse_cmdline(fdt); + parse_cmdline(fdt, chosen); for (i = 0; i < ARRAY_SIZE(regs); i++) { reg = prel64_to_pointer(®s[i].reg_prel); diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index eeecee7ffd6f..0257b43819db 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -16,17 +16,17 @@ #include #include +#include "pi.h" + extern u16 memstart_offset_seed; -static u64 __init get_kaslr_seed(void *fdt) +static u64 __init get_kaslr_seed(void *fdt, int node) { - static char const chosen_str[] __initconst = "chosen"; static char const seed_str[] __initconst = "kaslr-seed"; - int node, len; fdt64_t *prop; u64 ret; + int len; - node = fdt_path_offset(fdt, chosen_str); if (node < 0) return 0; @@ -39,14 +39,14 @@ static u64 __init get_kaslr_seed(void *fdt) return ret; } -asmlinkage u64 __init kaslr_early_init(void *fdt) +u64 __init kaslr_early_init(void *fdt, int chosen) { u64 seed, range; if (kaslr_disabled_cmdline()) return 0; - seed = get_kaslr_seed(fdt); + seed = get_kaslr_seed(fdt, chosen); if (!seed) { if (!__early_cpu_has_rndr() || !__arm64_rndr((unsigned long *)&seed)) diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c new file mode 100644 index 000000000000..9e64c14c1598 --- /dev/null +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -0,0 +1,165 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Author: Ard Biesheuvel + +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include "pi.h" + +extern const u8 __eh_frame_start[], __eh_frame_end[]; + +extern void idmap_cpu_replace_ttbr1(void *pgdir); + +static void map_segment(pgd_t *pg_dir, u64 *pgd, u64 va_offset, + void *start, void *end, pgprot_t prot, + bool may_use_cont, int root_level) +{ + map_range(pgd, ((u64)start + va_offset) & ~PAGE_OFFSET, + ((u64)end + va_offset) & ~PAGE_OFFSET, (u64)start, + prot, root_level, (pte_t *)pg_dir, may_use_cont, 0); +} + +static void unmap_segment(pgd_t *pg_dir, u64 va_offset, void *start, + void *end, int root_level) +{ + map_segment(pg_dir, NULL, va_offset, start, end, __pgprot(0), + false, root_level); +} + +static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) +{ + bool enable_scs = IS_ENABLED(CONFIG_UNWIND_PATCH_PAC_INTO_SCS); + bool twopass = IS_ENABLED(CONFIG_RELOCATABLE); + u64 pgdp = (u64)init_pg_dir + PAGE_SIZE; + pgprot_t text_prot = PAGE_KERNEL_ROX; + pgprot_t data_prot = PAGE_KERNEL; + pgprot_t prot; + + /* + * External debuggers may need to write directly to the text mapping to + * install SW breakpoints. Allow this (only) when explicitly requested + * with rodata=off. + */ + if (cpuid_feature_extract_unsigned_field(arm64_sw_feature_override.val, + ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF)) + text_prot = PAGE_KERNEL_EXEC; + + /* + * We only enable the shadow call stack dynamically if we are running + * on a system that does not implement PAC or BTI. PAC and SCS provide + * roughly the same level of protection, and BTI relies on the PACIASP + * instructions serving as landing pads, preventing us from patching + * those instructions into something else. + */ + if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL) && cpu_has_pac()) + enable_scs = false; + + if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL) && cpu_has_bti()) { + enable_scs = false; + + /* + * If we have a CPU that supports BTI and a kernel built for + * BTI then mark the kernel executable text as guarded pages + * now so we don't have to rewrite the page tables later. + */ + text_prot = __pgprot_modify(text_prot, PTE_GP, PTE_GP); + } + + /* Map all code read-write on the first pass if needed */ + twopass |= enable_scs; + prot = twopass ? data_prot : text_prot; + + map_segment(init_pg_dir, &pgdp, va_offset, _stext, _etext, prot, + !twopass, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, __start_rodata, + __inittext_begin, data_prot, false, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, __inittext_begin, + __inittext_end, prot, false, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, __initdata_begin, + __initdata_end, data_prot, false, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, _data, _end, data_prot, + true, root_level); + dsb(ishst); + + idmap_cpu_replace_ttbr1(init_pg_dir); + + if (twopass) { + if (IS_ENABLED(CONFIG_RELOCATABLE)) + relocate_kernel(kaslr_offset); + + if (enable_scs) { + scs_patch(__eh_frame_start + va_offset, + __eh_frame_end - __eh_frame_start); + asm("ic ialluis"); + + dynamic_scs_is_enabled = true; + } + + /* + * Unmap the text region before remapping it, to avoid + * potential TLB conflicts when creating the contiguous + * descriptors. + */ + unmap_segment(init_pg_dir, va_offset, _stext, _etext, + root_level); + dsb(ishst); + isb(); + __tlbi(vmalle1); + isb(); + + /* + * Remap these segments with different permissions + * No new page table allocations should be needed + */ + map_segment(init_pg_dir, NULL, va_offset, _stext, _etext, + text_prot, true, root_level); + map_segment(init_pg_dir, NULL, va_offset, __inittext_begin, + __inittext_end, text_prot, false, root_level); + dsb(ishst); + } +} + +asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) +{ + static char const chosen_str[] __initconst = "/chosen"; + u64 va_base, pa_base = (u64)&_text; + u64 kaslr_offset = pa_base % MIN_KIMG_ALIGN; + int root_level = 4 - CONFIG_PGTABLE_LEVELS; + int chosen; + + /* Clear BSS and the initial page tables */ + memset(__bss_start, 0, (u64)init_pg_end - (u64)__bss_start); + + /* Parse the command line for CPU feature overrides */ + chosen = fdt_path_offset(fdt, chosen_str); + init_feature_override(boot_status, fdt, chosen); + + /* + * The virtual KASLR displacement modulo 2MiB is decided by the + * physical placement of the image, as otherwise, we might not be able + * to create the early kernel mapping using 2 MiB block descriptors. So + * take the low bits of the KASLR offset from the physical address, and + * fill in the high bits from the seed. + */ + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) { + u64 kaslr_seed = kaslr_early_init(fdt, chosen); + + if (kaslr_seed && kaslr_requires_kpti()) + arm64_use_ng_mappings = true; + + kaslr_offset |= kaslr_seed & ~(MIN_KIMG_ALIGN - 1); + } + + va_base = KIMAGE_VADDR + kaslr_offset; + map_kernel(kaslr_offset, va_base - pa_base, root_level); +} diff --git a/arch/arm64/kernel/pi/map_range.c b/arch/arm64/kernel/pi/map_range.c new file mode 100644 index 000000000000..c31feda18f47 --- /dev/null +++ b/arch/arm64/kernel/pi/map_range.c @@ -0,0 +1,88 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Author: Ard Biesheuvel + +#include +#include + +#include +#include +#include + +#include "pi.h" + +/** + * map_range - Map a contiguous range of physical pages into virtual memory + * + * @pte: Address of physical pointer to array of pages to + * allocate page tables from + * @start: Virtual address of the start of the range + * @end: Virtual address of the end of the range (exclusive) + * @pa: Physical address of the start of the range + * @prot: Access permissions of the range + * @level: Translation level for the mapping + * @tbl: The level @level page table to create the mappings in + * @may_use_cont: Whether the use of the contiguous attribute is allowed + * @va_offset: Offset between a physical page and its current mapping + * in the VA space + */ +void __init map_range(u64 *pte, u64 start, u64 end, u64 pa, pgprot_t prot, + int level, pte_t *tbl, bool may_use_cont, u64 va_offset) +{ + u64 cmask = (level == 3) ? CONT_PTE_SIZE - 1 : U64_MAX; + u64 protval = pgprot_val(prot) & ~PTE_TYPE_MASK; + int lshift = (3 - level) * (PAGE_SHIFT - 3); + u64 lmask = (PAGE_SIZE << lshift) - 1; + + start &= PAGE_MASK; + pa &= PAGE_MASK; + + /* Advance tbl to the entry that covers start */ + tbl += (start >> (lshift + PAGE_SHIFT)) % PTRS_PER_PTE; + + /* + * Set the right block/page bits for this level unless we are + * clearing the mapping + */ + if (protval) + protval |= (level < 3) ? PMD_TYPE_SECT : PTE_TYPE_PAGE; + + while (start < end) { + u64 next = min((start | lmask) + 1, PAGE_ALIGN(end)); + + if (level < 3 && (start | next | pa) & lmask) { + /* + * This chunk needs a finer grained mapping. Create a + * table mapping if necessary and recurse. + */ + if (pte_none(*tbl)) { + *tbl = __pte(__phys_to_pte_val(*pte) | + PMD_TYPE_TABLE | PMD_TABLE_UXN); + *pte += PTRS_PER_PTE * sizeof(pte_t); + } + map_range(pte, start, next, pa, prot, level + 1, + (pte_t *)(__pte_to_phys(*tbl) + va_offset), + may_use_cont, va_offset); + } else { + /* + * Start a contiguous range if start and pa are + * suitably aligned + */ + if (((start | pa) & cmask) == 0 && may_use_cont) + protval |= PTE_CONT; + + /* + * Clear the contiguous attribute if the remaining + * range does not cover a contiguous block + */ + if ((end & ~cmask) <= start) + protval &= ~PTE_CONT; + + /* Put down a block or page mapping */ + *tbl = __pte(__phys_to_pte_val(pa) | protval); + } + pa += next - start; + start = next; + tbl++; + } +} diff --git a/arch/arm64/kernel/pi/patch-scs.c b/arch/arm64/kernel/pi/patch-scs.c index c65ef40d1e6b..49d8b40e61bc 100644 --- a/arch/arm64/kernel/pi/patch-scs.c +++ b/arch/arm64/kernel/pi/patch-scs.c @@ -11,6 +11,10 @@ #include +#include "pi.h" + +bool dynamic_scs_is_enabled; + // // This minimal DWARF CFI parser is partially based on the code in // arch/arc/kernel/unwind.c, and on the document below: @@ -46,8 +50,6 @@ #define DW_CFA_GNU_negative_offset_extended 0x2f #define DW_CFA_hi_user 0x3f -extern const u8 __eh_frame_start[], __eh_frame_end[]; - enum { PACIASP = 0xd503233f, AUTIASP = 0xd50323bf, @@ -250,13 +252,3 @@ int scs_patch(const u8 eh_frame[], int size) } return 0; } - -asmlinkage void __init scs_patch_vmlinux(const u8 start[], const u8 end[]) -{ - if (!should_patch_pac_into_scs()) - return; - - scs_patch(start, end - start); - asm("ic ialluis"); - isb(); -} diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h index f455ad385976..dfc70828ad0a 100644 --- a/arch/arm64/kernel/pi/pi.h +++ b/arch/arm64/kernel/pi/pi.h @@ -2,6 +2,8 @@ // Copyright 2023 Google LLC // Author: Ard Biesheuvel +#include + #define __prel64_initconst __section(".init.rodata.prel64") typedef volatile signed long prel64_t; @@ -12,3 +14,13 @@ static inline void *prel64_to_pointer(const prel64_t *offset) return NULL; return (void *)offset + *offset; } + +extern bool dynamic_scs_is_enabled; + +void init_feature_override(u64 boot_status, const void *fdt, int chosen); +u64 kaslr_early_init(void *fdt, int chosen); +void relocate_kernel(u64 offset); +int scs_patch(const u8 eh_frame[], int size); + +void map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, + int level, pte_t *tbl, bool may_use_cont, u64 va_offset); diff --git a/arch/arm64/kernel/pi/relocate.c b/arch/arm64/kernel/pi/relocate.c index 1853408ea76b..2407d2696398 100644 --- a/arch/arm64/kernel/pi/relocate.c +++ b/arch/arm64/kernel/pi/relocate.c @@ -7,6 +7,8 @@ #include #include +#include "pi.h" + extern const Elf64_Rela rela_start[], rela_end[]; extern const u64 relr_start[], relr_end[]; diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index c2d6852a4e0c..f75e03b9ad28 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -283,13 +283,6 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) kaslr_init(); - /* - * If know now we are going to need KPTI then use non-global - * mappings from the start, avoiding the cost of rewriting - * everything later. - */ - arm64_use_ng_mappings = kaslr_enabled() && kaslr_requires_kpti(); - early_fixmap_init(); early_ioremap_init(); diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 3afb4223a5e8..755a22d4f840 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -126,9 +126,9 @@ jiffies = jiffies_64; #ifdef CONFIG_UNWIND_TABLES #define UNWIND_DATA_SECTIONS \ .eh_frame : { \ - __eh_frame_start = .; \ + __pi___eh_frame_start = .; \ *(.eh_frame) \ - __eh_frame_end = .; \ + __pi___eh_frame_end = .; \ } #else #define UNWIND_DATA_SECTIONS diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 14fdf645edc8..30b066ac4a74 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -195,6 +195,7 @@ SYM_TYPED_FUNC_START(idmap_cpu_replace_ttbr1) ret SYM_FUNC_END(idmap_cpu_replace_ttbr1) +SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .popsection #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 From patchwork Tue Sep 12 14:16:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381795 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D2523CA0EEB for ; Tue, 12 Sep 2023 14:20:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=+ZzFqHdGY7MO0J7Bs2fZJDCvlWvvuLg3ZuVXCVCDJxM=; b=qtRITNxYP6kpMrbUkA8gw0wevy xbrH+mzVVawVNhI6WI/fSqPEZKsg3MhcQc+8tdesSSXV3vWunKiP2Ur2s7gJVf99NtUc8XAgkMmy0 8GxR+Ucdbw/+5obU1AmQjLC5J8dTZiAu6N+RaFScXQJrf6dRlEYc6JSUzjnH4XhA5mfOGg2DCrH+E jxkAvAUaJjZfQm8T48JC3A8cU9Hrb7bOA57NnQ/XBPlJDIINDjCbUi+8rCDcdnaLjkCeHD8X4Gvsy 6Ecy6/0E+jQNBCYC03U/pp1m56l9BEnrE4uQlrjK26M9nY3CFEUmHxGLC4EspbfW/qjoZ+1Kb0633 EbciaNow==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4El-003XXr-0v; Tue, 12 Sep 2023 14:19:31 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4E6-003WyI-2X for linux-arm-kernel@bombadil.infradead.org; Tue, 12 Sep 2023 14:18:50 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=CaXnLCEyTKksb5yUJ9O1Yp+FnZR4E/gfzPnjUBp/gac=; b=pZntRfpOFTcmlreD2juyBVnXKO LsiughPV41Iv4aobfuCAg8Xc87EgupnSxqMtCWro+KdJRLVoZONCPBnrE1nfX7p+YIRvzg2AuGAIx 89juKdABQz1WrIrwd+smLDlGd7xCeR66BLob2QNPqjnoJGp2QKFXws/R80YChHzmPyuRrxmUeK3xC s5TK6yb3AFHP9s6ZcuwjXgmhxcBx8hG/QKg5rBwtt3kXFd842XWYCyNOt9mc4xQzcboNTwaUtFS+O 1e5oLkFG5bV8L9p1iqVduzTdkjjek9xty/oPJblinuzz2+bjdQL+MRbMcRlPC0YP2rIJJ2Wq+RRL9 HdaioTcg==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Dz-0069zS-09 for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:49 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-59b5a586da6so77311247b3.1 for ; Tue, 12 Sep 2023 07:18:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528319; x=1695133119; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=CaXnLCEyTKksb5yUJ9O1Yp+FnZR4E/gfzPnjUBp/gac=; b=aU7TWTKk70PXEqnGaGYxfUDtaxOp8aBZnH5U4t6h/Yaw+XnPg87cKQr+0vt0IVVqxy NcVX5aXONjj41BVC32vzjOxOqGR5dejkFtV4uzGd32E3x/SfbviHemHDxss5BiJ2/ngu FnAVixNMuWfC173R88F/gMFSdxeqMWYcnoZW9VKV6kJsZwoqKJG2AxzVpDPJHXh2rH2G zi8oMnlaj5rPHL2vIzRb/YNKj+fzryfP3yK2QBjt/bSSioJ6dfGRMI7mRXgHFANeu8QU w63BVcpEfeGwHHe0HjoEz1pOnrBFwJZC9Cv564aR1zvUhUD2SckvKZ4lWCGf8Dx2aTEZ 0yCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528319; x=1695133119; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=CaXnLCEyTKksb5yUJ9O1Yp+FnZR4E/gfzPnjUBp/gac=; b=RZb7U3UD6bCH2qPTHmfMZGx4jOM2hNYtTcDytftnnYAEDJ8P5tOD1/VWqTwgu3ltv5 Ib+S9kdT3RlEgsnKCa6riHBl79zRXMLvKb9m1BPtCpG+o7G2HTIGUHHoYBWlzRhu/SwC pn973NeVuO7+3XXsb82Pq6Zb/+VHdz6jzI3xTgMv19TOluO6Z8maPOAsuDXClrx72VGN nlmtnqmc7wp1f94Usn+eBE6Be8fb+wT1UN8GspToj664vTNQHA0RgkTuKHtXDFBD9cBu T1BOYb29+zI2e9os1noWDD1TG2qcwFxGYpiphVQ16z/r12N4xDfSR1hweJLlMOY6b9eZ 5i7Q== X-Gm-Message-State: AOJu0YwgAb5qrvjyK810T2ijpHB/oHo2BCP9qER5xgmb+EPRn4KTKuvC Kx7SWvuN5TXD6FdAo/xIWqvC2zEJVhR1aLGV9jJRfmdr2pqbRSmEiQ41ok1cr56PTXkLMwiozZz H5Y6vGwNlFVHAk8sLGTVqup204g8gIf1kgmIOM86C6ptI/5Kacx+5HSlNaDwJMIfhcoiiV7TWLz s= X-Google-Smtp-Source: AGHT+IHIzOi2TH8IzZmJtvv++8yzvrmhQ9NVkVC2PZpcNQT/t7fqzKWKhPACavKEIovW73q/tJUaRNkh X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:690c:311:b0:59b:5a5b:3a91 with SMTP id bg17-20020a05690c031100b0059b5a5b3a91mr95062ywb.2.1694528318837; Tue, 12 Sep 2023 07:18:38 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:24 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=7010; i=ardb@kernel.org; h=from:subject; bh=IP1ASo65b8oOf3V6YkSoNzvt+dQQ7S6GDOamX919CTg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaOfMxFM+c+Vi4oIi+ZfZx3LeVs7Td5gmxtl1seH+t vf3+RQ7SlkYxDgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwEQe5TAynOA85yv9Kq6N83/X LN5qiw9LHf/tzL/YJPTgq6jywz2BkxgZrk85NWXVo6XGkxsfK3uE+bY1f99Q1xl+JSK/8ET+3N1 z2AE= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-97-ardb@google.com> Subject: [PATCH v4 34/61] arm64: mm: Use 48-bit virtual addressing for the permanent ID map From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_151844_608969_0DFD05D1 X-CRM114-Status: GOOD ( 25.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Even though we support loading kernels anywhere in 48-bit addressable physical memory, we create the ID maps based on the number of levels that we happened to configure for the kernel VA and user VA spaces. The reason for this is that the PGD/PUD/PMD based classification of translation levels, along with the associated folding when the number of levels is less than 5, does not permit creating a page table hierarchy of a set number of levels. This means that, for instance, on 39-bit VA kernels we need to configure an additional level above PGD level on the fly, and 36-bit VA kernels still only support 47-bit virtual addressing with this trick applied. Now that we have a separate helper to populate page table hierarchies that does not define the levels in terms of PUDS/PMDS/etc at all, let's reuse it to create the permanent ID map with a fixed VA size of 48 bits. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 3 ++ arch/arm64/kernel/head.S | 5 +++ arch/arm64/kvm/mmu.c | 15 +++------ arch/arm64/mm/mmu.c | 32 +++++++++++--------- arch/arm64/mm/proc.S | 9 ++---- 5 files changed, 32 insertions(+), 32 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 0631604995ee..742a4b2778f7 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -35,6 +35,9 @@ #define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS) #endif +#define IDMAP_VA_BITS 48 +#define IDMAP_LEVELS ARM64_HW_PGTABLE_LEVELS(IDMAP_VA_BITS) +#define IDMAP_ROOT_LEVEL (4 - IDMAP_LEVELS) /* * A relocatable kernel may execute from an address that differs from the one at diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index ffacce7b5a02..a1c29d64e875 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -729,6 +729,11 @@ SYM_FUNC_START_LOCAL(__no_granule_support) SYM_FUNC_END(__no_granule_support) SYM_FUNC_START_LOCAL(__primary_switch) + mrs x1, tcr_el1 + mov x2, #64 - VA_BITS + tcr_set_t0sz x1, x2 + msr tcr_el1, x1 + adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 587a104f66c3..beee6408534d 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1889,16 +1889,9 @@ int __init kvm_mmu_init(u32 *hyp_va_bits) BUG_ON((hyp_idmap_start ^ (hyp_idmap_end - 1)) & PAGE_MASK); /* - * The ID map may be configured to use an extended virtual address - * range. This is only the case if system RAM is out of range for the - * currently configured page size and VA_BITS_MIN, in which case we will - * also need the extended virtual range for the HYP ID map, or we won't - * be able to enable the EL2 MMU. - * - * However, in some cases the ID map may be configured for fewer than - * the number of VA bits used by the regular kernel stage 1. This - * happens when VA_BITS=52 and the kernel image is placed in PA space - * below 48 bits. + * The ID map is always configured for 48 bits of translation, which + * may be fewer than the number of VA bits used by the regular kernel + * stage 1, when VA_BITS=52. * * At EL2, there is only one TTBR register, and we can't switch between * translation tables *and* update TCR_EL2.T0SZ at the same time. Bottom @@ -1909,7 +1902,7 @@ int __init kvm_mmu_init(u32 *hyp_va_bits) * 1 VA bits to assure that the hypervisor can both ID map its code page * and map any kernel memory. */ - idmap_bits = 64 - ((idmap_t0sz & TCR_T0SZ_MASK) >> TCR_T0SZ_OFFSET); + idmap_bits = IDMAP_VA_BITS; kernel_bits = vabits_actual; *hyp_va_bits = max(idmap_bits, kernel_bits); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 49a49b37580b..070bc1bc5ad1 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -754,22 +754,21 @@ static void __init map_kernel(pgd_t *pgdp) kasan_copy_shadow(pgdp); } +void __pi_map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, + int level, pte_t *tbl, bool may_use_cont, u64 va_offset); + +static u8 idmap_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init, + kpti_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_init; + static void __init create_idmap(void) { u64 start = __pa_symbol(__idmap_text_start); - u64 size = __pa_symbol(__idmap_text_end) - start; - pgd_t *pgd = idmap_pg_dir; - u64 pgd_phys; - - /* check if we need an additional level of translation */ - if (VA_BITS < 48 && idmap_t0sz < (64 - VA_BITS_MIN)) { - pgd_phys = early_pgtable_alloc(PAGE_SHIFT); - set_pgd(&idmap_pg_dir[start >> VA_BITS], - __pgd(pgd_phys | P4D_TYPE_TABLE)); - pgd = __va(pgd_phys); - } - __create_pgd_mapping(pgd, start, start, size, PAGE_KERNEL_ROX, - early_pgtable_alloc, 0); + u64 end = __pa_symbol(__idmap_text_end); + u64 ptep = __pa_symbol(idmap_ptes); + + __pi_map_range(&ptep, start, end, start, PAGE_KERNEL_ROX, + IDMAP_ROOT_LEVEL, (pte_t *)idmap_pg_dir, false, + __phys_to_virt(ptep) - ptep); if (IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0)) { extern u32 __idmap_kpti_flag; @@ -779,8 +778,10 @@ static void __init create_idmap(void) * The KPTI G-to-nG conversion code needs a read-write mapping * of its synchronization flag in the ID map. */ - __create_pgd_mapping(pgd, pa, pa, sizeof(u32), PAGE_KERNEL, - early_pgtable_alloc, 0); + ptep = __pa_symbol(kpti_ptes); + __pi_map_range(&ptep, pa, pa + sizeof(u32), pa, PAGE_KERNEL, + IDMAP_ROOT_LEVEL, (pte_t *)idmap_pg_dir, false, + __phys_to_virt(ptep) - ptep); } } @@ -805,6 +806,7 @@ void __init paging_init(void) memblock_allow_resize(); create_idmap(); + idmap_t0sz = TCR_T0SZ(IDMAP_VA_BITS); } #ifdef CONFIG_MEMORY_HOTPLUG diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 30b066ac4a74..8432af63e023 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -422,9 +422,9 @@ SYM_FUNC_START(__cpu_setup) mair .req x17 tcr .req x16 mov_q mair, MAIR_EL1_SET - mov_q tcr, TCR_TxSZ(VA_BITS) | TCR_CACHE_FLAGS | TCR_SMP_FLAGS | \ - TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ - TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS + mov_q tcr, TCR_T0SZ(IDMAP_VA_BITS) | TCR_T1SZ(VA_BITS) | TCR_CACHE_FLAGS | \ + TCR_SMP_FLAGS | TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ + TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS tcr_clear_errata_bits tcr, x9, x5 @@ -432,10 +432,7 @@ SYM_FUNC_START(__cpu_setup) sub x9, xzr, x0 add x9, x9, #64 tcr_set_t1sz tcr, x9 -#else - idmap_get_t0sz x9 #endif - tcr_set_t0sz tcr, x9 /* * Set the IPS bits in TCR_EL1. From patchwork Tue Sep 12 14:16:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381796 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1C6D7CA0EEC for ; Tue, 12 Sep 2023 14:20:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=G7RgftsZu0sa9jLZvNOu3eQlDi0SGKjyYr1vC5o5SeY=; b=3k7EXpRHRRqsgCMRkyaDKbarq2 Q4tlk5a5JILbymGXMDe/+dsUOcw7Num9u9HlV93KFGPvLTF2d5fn2qXqDZP09Sgn3NZd4f74MZomn a72/i40PzLViPiQAWUIKDNTjjF/XkZPrN9zO3D+NIIDBxB1sR0iP+2FXXdZPlj4zOYitR3we6pdxz Bk/7P6zr3tkETJuVzd6oJ3aenJ+7+Y47EliGMH//R/TsGwrzjtkoDzxlsXzlCxFwbmIiJyUJoHyQy 7dwwV258cAO7V8//My/6F1dUFzENRZgKy8ktCUS8LAmTHvaDtfGP8TjfEIwpDDUN8LyxSF08tfQbZ 3v48gyQA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Ej-003XWC-05; Tue, 12 Sep 2023 14:19:29 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4E4-003WwF-1b for linux-arm-kernel@bombadil.infradead.org; Tue, 12 Sep 2023 14:18:48 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=7rSPmb28NDHeLvOcEanQajPSW8hxzl+nLXq6JrjyI9s=; b=kecX8OLjV/3yo3g/YCIXDW3/LV PvQ9AE2LjzAaz5fJ/5IQMMNfPOT+c65OxPvpqV8GcZfosw0hnUrMnnz18puOV5GQ86PjkALAu5B8R QXgC4hXn/66KfVFdd+FFNkt8xxKykLMeCB/uFWJdGCMaJL3SKsoHmShar21tBKRlzzHbdkC6L+pCY wmA9qOo6Hf2S+gihDvXUxAwcC6y/mcE+QP7t3SpSAuCwpevSQpCv/OmQodu+EeCXN9QNHEwelSQze 2UwRUTPbG4Wf464FFdMLFl3ucc1CijfMI3IUe2TCCNPKvEJ91tRKLeSnUtdD34Ur09AMy9luVoDe8 NIXRK/tw==; Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Dy-0069zW-0g for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:46 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-31c5cc3b512so3704908f8f.1 for ; Tue, 12 Sep 2023 07:18:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528321; x=1695133121; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7rSPmb28NDHeLvOcEanQajPSW8hxzl+nLXq6JrjyI9s=; b=Ia21qUluD9oiKO86SqenezasnEjn473/XEMRfNRD9AZvFUlIKAxX+dV8+kUqeKOHsh W6WxUA3uoKhv+pzc/K5ZwUh4HWbigOEw4i9+ge6VW0+Uw/RMvkhsCRDhhac4jR6djT5N JAVPcvsLSWq5WSNcLoStO2V72mz77XiNAOZbaL7RZPfCSnyCFqo2KD69XB8xNvkmX+cK ybQL8BeMBDz6Pjm24aGWdyTqhSopn8gqM+iWz6cKZXbwbP7Iyslcy4cB3J9MiczO4PPG jVHC2Thb+q9rHOgbKJjDR2SsOyPqkQ7qaPq+yJwb0GtHb/zMlG/lVYMIyZhkZw6S9nM0 jLMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528321; x=1695133121; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7rSPmb28NDHeLvOcEanQajPSW8hxzl+nLXq6JrjyI9s=; b=k4gkkFcN3o86TQp5a9XrYSE+5bwm1on40MLp5SGEm80lqwPWfHRXINtlk3I0h2HjKu XfD5bI8haL+Cb1Ck1HIipvFcaxXzohDYuU7/rEoFUFGFLDNrqGB4dsj2f/hwjJFDVCnh +LlR4TuSBaKkc8lsj2lff+7bHbCHKqdH4vPOYBcsf3NrW0gZpBFiaEugDvpNDTqt07FA Mzvq8gJpSVql39Gt01tO1agMj1dv42IkEOKIfzYy6/uGprpIH0JmiPGjL4qd69oSxXoY mRtzsAChVgrwah2tn/mWSAhIg2O1CfPhw30aZKniszomy1Xgm/d1Zej/wZ4VmeFFZHzI 8kWw== X-Gm-Message-State: AOJu0Yz3MhpHmEqh+Gdorjze5YdqALHYyE2iSgt2ShGXKy1Vsu52sCwN 7Ysujqz65agwP3JRlqDtGh521W1jWr3ARRPwvaCt4jpsYbqlKBo8NIlawm07KVGMXq66S3fcPHy ps1DBdSfAGtwKIdHJEgEXcf32t01rhAGp32WafN/aLxMPQgTdC2DUlzON1WWmZzhyeTMPUb+WjB I= X-Google-Smtp-Source: AGHT+IG0zayvnQdH/RqoUrJoPRXJEIJmOddCZXS7OJVSGPZyfMKc4D+VRvRtz5OhI7lJ6PX/DA6pZ3h7 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:60cb:0:b0:317:690e:7247 with SMTP id x11-20020a5d60cb000000b00317690e7247mr166538wrt.0.1694528321037; Tue, 12 Sep 2023 07:18:41 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:25 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5162; i=ardb@kernel.org; h=from:subject; bh=yFqAuJnKNAsURVmbFl/LbDGXxKPczokDU1lCacgcPzM=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaFda8JckW17PZR1zeDgku/uOJ9Roh5w94ytlftH4l MazD/c6SlkYxDgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwERkTRj+R/96Kbpxu/WhRbeP 1aow6U+RceH/ayShmRmRztw8M+ZdDcM/W+44GdvF+UzHsyJOtq72a5V7N/tQwrRuH88PR+beO8j CAAA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-98-ardb@google.com> Subject: [PATCH v4 35/61] arm64: pgtable: Decouple PGDIR size macros from PGD/PUD/PMD levels From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_151843_776117_6B220D73 X-CRM114-Status: GOOD ( 19.67 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The mapping from PGD/PUD/PMD to levels and shifts is very confusing, given that, due to folding, the shifts may be equal for different levels, if the macros are even #define'd to begin with. In a subsequent patch, we will modify the ID mapping code to decouple the number of levels from the kernel's view of how these types are folded, so prepare for this by reformulating the macros without the use of these types. Instead, use SWAPPER_BLOCK_SHIFT as the base quantity, and derive it from either PAGE_SHIFT or PMD_SHIFT, which -if defined at all- are defined unambiguously for a given page size, regardless of the number of configured levels. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 65 ++++++-------------- 1 file changed, 19 insertions(+), 46 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 742a4b2778f7..5000f38ae0c6 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -13,27 +13,22 @@ #include /* - * The linear mapping and the start of memory are both 2M aligned (per - * the arm64 booting.txt requirements). Hence we can use section mapping - * with 4K (section size = 2M) but not with 16K (section size = 32M) or - * 64K (section size = 512M). + * The physical and virtual addresses of the start of the kernel image are + * equal modulo 2 MiB (per the arm64 booting.txt requirements). Hence we can + * use section mapping with 4K (section size = 2M) but not with 16K (section + * size = 32M) or 64K (section size = 512M). */ - -/* - * The idmap and swapper page tables need some space reserved in the kernel - * image. Both require pgd, pud (4 levels only) and pmd tables to (section) - * map the kernel. With the 64K page configuration, swapper and idmap need to - * map to pte level. The swapper also maps the FDT (see __create_page_tables - * for more information). Note that the number of ID map translation levels - * could be increased on the fly if system RAM is out of reach for the default - * VA range, so pages required to map highest possible PA are reserved in all - * cases. - */ -#ifdef CONFIG_ARM64_4K_PAGES -#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - 1) +#if defined(PMD_SIZE) && PMD_SIZE <= MIN_KIMG_ALIGN +#define SWAPPER_BLOCK_SHIFT PMD_SHIFT +#define SWAPPER_SKIP_LEVEL 1 #else -#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS) +#define SWAPPER_BLOCK_SHIFT PAGE_SHIFT +#define SWAPPER_SKIP_LEVEL 0 #endif +#define SWAPPER_BLOCK_SIZE (UL(1) << SWAPPER_BLOCK_SHIFT) +#define SWAPPER_TABLE_SHIFT (SWAPPER_BLOCK_SHIFT + PAGE_SHIFT - 3) + +#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - SWAPPER_SKIP_LEVEL) #define IDMAP_VA_BITS 48 #define IDMAP_LEVELS ARM64_HW_PGTABLE_LEVELS(IDMAP_VA_BITS) @@ -53,24 +48,13 @@ #define EARLY_ENTRIES(vstart, vend, shift, add) \ (SPAN_NR_ENTRIES(vstart, vend, shift) + (add)) -#define EARLY_PGDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, PGDIR_SHIFT, add)) - -#if SWAPPER_PGTABLE_LEVELS > 3 -#define EARLY_PUDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, PUD_SHIFT, add)) -#else -#define EARLY_PUDS(vstart, vend, add) (0) -#endif +#define EARLY_LEVEL(l, vstart, vend, add) \ + (SWAPPER_PGTABLE_LEVELS > l ? EARLY_ENTRIES(vstart, vend, SWAPPER_BLOCK_SHIFT + l * (PAGE_SHIFT - 3), add) : 0) -#if SWAPPER_PGTABLE_LEVELS > 2 -#define EARLY_PMDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, SWAPPER_TABLE_SHIFT, add)) -#else -#define EARLY_PMDS(vstart, vend, add) (0) -#endif - -#define EARLY_PAGES(vstart, vend, add) ( 1 /* PGDIR page */ \ - + EARLY_PGDS((vstart), (vend), add) /* each PGDIR needs a next level page table */ \ - + EARLY_PUDS((vstart), (vend), add) /* each PUD needs a next level page table */ \ - + EARLY_PMDS((vstart), (vend), add)) /* each PMD needs a next level page table */ +#define EARLY_PAGES(vstart, vend, add) (1 /* PGDIR page */ \ + + EARLY_LEVEL(3, (vstart), (vend), add) /* each entry needs a next level page table */ \ + + EARLY_LEVEL(2, (vstart), (vend), add) /* each entry needs a next level page table */ \ + + EARLY_LEVEL(1, (vstart), (vend), add))/* each entry needs a next level page table */ #define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE) + EARLY_SEGMENT_EXTRA_PAGES)) /* the initial ID map may need two extra pages if it needs to be extended */ @@ -81,17 +65,6 @@ #endif #define INIT_IDMAP_DIR_PAGES EARLY_PAGES(KIMAGE_VADDR, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE, 1) -/* Initial memory map size */ -#ifdef CONFIG_ARM64_4K_PAGES -#define SWAPPER_BLOCK_SHIFT PMD_SHIFT -#define SWAPPER_BLOCK_SIZE PMD_SIZE -#define SWAPPER_TABLE_SHIFT PUD_SHIFT -#else -#define SWAPPER_BLOCK_SHIFT PAGE_SHIFT -#define SWAPPER_BLOCK_SIZE PAGE_SIZE -#define SWAPPER_TABLE_SHIFT PMD_SHIFT -#endif - /* The number of segments in the kernel image (text, rodata, inittext, initdata, data+bss) */ #define KERNEL_SEGMENT_COUNT 5 From patchwork Tue Sep 12 14:16:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E20D6CA0EEB for ; Tue, 12 Sep 2023 14:20:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=g6kERK2lrapjxeKEGf+/dZCoL2Tt5xzwFJAVatdTtqA=; b=VydB5DiRhtQLcLyfJKc02TwSh/ ZSRfULLYvi8Dmc2Z8huJyw+sLYhSH3TpVW66MaV53zs3xIm9rvLVGSEw4XcyepAdqS0wvAdKbJzgH BramvnaOEWyBGelAgRg030jl1VuOT/c5T1jdxWcKHHZY6/jqdb2TQg94ASVvZxR2/WjjhQFSs8t8Z sGVLwr6mL6z0wZQjxKgRff1Pfn0Bwj6vMbC7gM6FHjjv48D8qWQGGYElj4ivURq+h2Q7LMLCYGp2t A2OrA6XgxaL35gO9dBw73JnKHW1nGspmRPZ16K8ysDZrn5Osqo5S55F3NmOs4QtQT4qaPLK3pseDB 6BJxAWig==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4F3-003Xm6-1O; Tue, 12 Sep 2023 14:19:49 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4E1-003Wt9-2O for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:02 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-d7e79ec07b4so5569829276.0 for ; Tue, 12 Sep 2023 07:18:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528324; x=1695133124; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=N/uLknh9f4OD6Z2c4jqlow8O0+stOukzMsUlzeMbFV8=; b=Q08zDTdW2ueB8Rq9nK0F7EtBQQumTFm1rGn9RhK0wF6zhayVbrsZJJdkirCo28LqkA xwFJ85eR56kyqbOMZzxY0WoEM4DRmS+ZYmCGJ6a/9E1fKBirCsD3VuPl7y5/SAXX6KVM 4/S7PbuvkEGcafPPKt/6MxuaZwlFnmnYI2PthbdVjPqUtjRBGP+2lS9WdkJ4snsgrY4V mTZ41RkM2G9vBiX4zyrOvoi+X706G5D+70fLkAgAd3skYGEpnzOsDCjZZ+eUdARIwzaU UCOIrgXqDLbqj+PmrEWht10iLnEP94OUeqz+/AV8rMzFqfv9WkLRqWI9zon6XibbNZpN /GUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528324; x=1695133124; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=N/uLknh9f4OD6Z2c4jqlow8O0+stOukzMsUlzeMbFV8=; b=qalRkk8NQ4WoPAQRx8SZwL0mo9g8MrLk1TnkDXyz3JN0RMJXW77oRAyUVy7jnBJgvN n8AxpUIjdZcBcxm9/CoxTMT8xTcHXM0/0Tw7m6OGA2V8srcJ0PaE7tpH2XyAEVGsfHsp dVpurWKOKEeqiFkD1sRTTJuuo/neFvcUbkMKtpJbmRI8rd4BpiqvOho3G2QVifZnZKFy Y24ISgRcEvCodvS4cTcVFh3WHYkmqOhWwIOXX3p1A9Y0c6Ft7wisc459xY5WpyUK8tI2 aWxO/TmT7IOQNNmZaKbZ/BNq7Oz+TuU3F22LlPDkzNHEtc/jpqUj2nyp7BiosnkCFtaN UHZQ== X-Gm-Message-State: AOJu0YxZf1zlz70AoNYkbk8lCcJ7IhBPIsElVw9xfaPRFIL4epieB194 mjyEp0E0Tww3+NjxoGSBa+/Gj2wovAaVGXOOfNYe1Pq3nsxsc5IU+s1pL8loi472kU2i7ljVJ2I KCxia/KjYHe4hXswqo/jTs2PewGVaOYWnpvcZ/8d5bczWJTTdgPvXoDpUZoAzX34g7RSn0UDKyQ s= X-Google-Smtp-Source: AGHT+IGy2081upfoWekcyLAi8yKjWCGFH8WcJt2h6R7cejH1P5YHw3h9/vqORbna+ABw2aPbQxjHXrCv X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:cd41:0:b0:d80:904d:c211 with SMTP id d62-20020a25cd41000000b00d80904dc211mr155806ybf.7.1694528323401; Tue, 12 Sep 2023 07:18:43 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:26 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=24438; i=ardb@kernel.org; h=from:subject; bh=mA2hw5XRTdzmN4gBlBVUvOA5WylCaeJ7byzsEaAHb0E=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaLeEqPKZ/ssm51+rfPhZlVBZYboluL82I+rWoYk1/ rXBkiodpSwMYhwMsmKKLAKz/77beXqiVK3zLFmYOaxMIEMYuDgFYCJ2RYwMe/t+ilZzmJhdnVHy PqbQ9rtLve97wQmfqxbJBb5tFFrZzfBXSExh4a43ZgJq/a17S1lqZ3QcZIneHf49f7/H5tnTg9J 5AQ== X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-99-ardb@google.com> Subject: [PATCH v4 36/61] arm64: kernel: Create initial ID map from C code From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071845_833518_BA3EE111 X-CRM114-Status: GOOD ( 33.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The asm code that creates the initial ID map is rather intricate and hard to follow. This is problematic because it makes adding support for things like LPA2 or WXN more difficult than necessary. Also, it is parameterized like the rest of the MM code to run with a configurable number of levels, which is rather pointless, given that all AArch64 CPUs implement support for 48-bit virtual addressing, and that many systems exist with DRAM located outside of the 39-bit addressable range, which is the only smaller VA size that is widely used, and we need additional tricks to make things work in that combination. So let's bite the bullet, and rip out all the asm macros, and fiddly code, and replace it with a C implementation based on the newly added routines for creating the early kernel VA mappings. And while at it, create the initial ID map based on 48-bit virtual addressing as well, regardless of the number of configured levels for the kernel proper. Note that this code may execute with the MMU and caches disabled, and is therefore not permitted to make unaligned accesses. This shouldn't generally happen in any case for the algorithm as implemented, but to be sure, let's pass -mstrict-align to the compiler just in case. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 14 - arch/arm64/include/asm/kernel-pgtable.h | 50 ++-- arch/arm64/include/asm/mmu_context.h | 6 +- arch/arm64/kernel/head.S | 267 ++------------------ arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/Makefile | 3 + arch/arm64/kernel/pi/map_kernel.c | 18 ++ arch/arm64/kernel/pi/map_range.c | 12 + arch/arm64/kernel/pi/pi.h | 4 + arch/arm64/mm/mmu.c | 5 - arch/arm64/mm/proc.S | 3 +- 11 files changed, 88 insertions(+), 295 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 376a980f2bad..beb53bbd8c19 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -345,20 +345,6 @@ alternative_cb_end bfi \valreg, \t1sz, #TCR_T1SZ_OFFSET, #TCR_TxSZ_WIDTH .endm -/* - * idmap_get_t0sz - get the T0SZ value needed to cover the ID map - * - * Calculate the maximum allowed value for TCR_EL1.T0SZ so that the - * entire ID map region can be mapped. As T0SZ == (64 - #bits used), - * this number conveniently equals the number of leading zeroes in - * the physical address of _end. - */ - .macro idmap_get_t0sz, reg - adrp \reg, _end - orr \reg, \reg, #(1 << VA_BITS_MIN) - 1 - clz \reg, \reg - .endm - /* * tcr_compute_pa_size - set TCR.(I)PS to the highest supported * ID_AA64MMFR0_EL1.PARange value diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 5000f38ae0c6..124a9f40008f 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -29,6 +29,7 @@ #define SWAPPER_TABLE_SHIFT (SWAPPER_BLOCK_SHIFT + PAGE_SHIFT - 3) #define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - SWAPPER_SKIP_LEVEL) +#define INIT_IDMAP_PGTABLE_LEVELS (IDMAP_LEVELS - SWAPPER_SKIP_LEVEL) #define IDMAP_VA_BITS 48 #define IDMAP_LEVELS ARM64_HW_PGTABLE_LEVELS(IDMAP_VA_BITS) @@ -48,44 +49,39 @@ #define EARLY_ENTRIES(vstart, vend, shift, add) \ (SPAN_NR_ENTRIES(vstart, vend, shift) + (add)) -#define EARLY_LEVEL(l, vstart, vend, add) \ - (SWAPPER_PGTABLE_LEVELS > l ? EARLY_ENTRIES(vstart, vend, SWAPPER_BLOCK_SHIFT + l * (PAGE_SHIFT - 3), add) : 0) +#define EARLY_LEVEL(l, lvls, vstart, vend, add) \ + (lvls > l ? EARLY_ENTRIES(vstart, vend, SWAPPER_BLOCK_SHIFT + l * (PAGE_SHIFT - 3), add) : 0) -#define EARLY_PAGES(vstart, vend, add) (1 /* PGDIR page */ \ - + EARLY_LEVEL(3, (vstart), (vend), add) /* each entry needs a next level page table */ \ - + EARLY_LEVEL(2, (vstart), (vend), add) /* each entry needs a next level page table */ \ - + EARLY_LEVEL(1, (vstart), (vend), add))/* each entry needs a next level page table */ -#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE) + EARLY_SEGMENT_EXTRA_PAGES)) +#define EARLY_PAGES(lvls, vstart, vend, add) (1 /* PGDIR page */ \ + + EARLY_LEVEL(3, (lvls), (vstart), (vend), add) /* each entry needs a next level page table */ \ + + EARLY_LEVEL(2, (lvls), (vstart), (vend), add) /* each entry needs a next level page table */ \ + + EARLY_LEVEL(1, (lvls), (vstart), (vend), add))/* each entry needs a next level page table */ +#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(SWAPPER_PGTABLE_LEVELS, KIMAGE_VADDR, _end, EXTRA_PAGE) \ + + EARLY_SEGMENT_EXTRA_PAGES)) -/* the initial ID map may need two extra pages if it needs to be extended */ -#if VA_BITS < 48 -#define INIT_IDMAP_DIR_SIZE ((INIT_IDMAP_DIR_PAGES + 2) * PAGE_SIZE) -#else -#define INIT_IDMAP_DIR_SIZE (INIT_IDMAP_DIR_PAGES * PAGE_SIZE) -#endif -#define INIT_IDMAP_DIR_PAGES EARLY_PAGES(KIMAGE_VADDR, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE, 1) +#define INIT_IDMAP_DIR_PAGES (EARLY_PAGES(INIT_IDMAP_PGTABLE_LEVELS, KIMAGE_VADDR, _end, 1)) +#define INIT_IDMAP_DIR_SIZE ((INIT_IDMAP_DIR_PAGES + EARLY_IDMAP_EXTRA_PAGES) * PAGE_SIZE) + +#define INIT_IDMAP_FDT_PAGES (EARLY_PAGES(INIT_IDMAP_PGTABLE_LEVELS, 0UL, UL(MAX_FDT_SIZE), 1) - 1) +#define INIT_IDMAP_FDT_SIZE ((INIT_IDMAP_FDT_PAGES + EARLY_IDMAP_EXTRA_FDT_PAGES) * PAGE_SIZE) /* The number of segments in the kernel image (text, rodata, inittext, initdata, data+bss) */ #define KERNEL_SEGMENT_COUNT 5 #if SWAPPER_BLOCK_SIZE > SEGMENT_ALIGN #define EARLY_SEGMENT_EXTRA_PAGES (KERNEL_SEGMENT_COUNT + 1) -#else -#define EARLY_SEGMENT_EXTRA_PAGES 0 -#endif - /* - * Initial memory map attributes. + * The initial ID map consists of the kernel image, mapped as two separate + * segments, and may appear misaligned wrt the swapper block size. This means + * we need 3 additional pages. The DT could straddle a swapper block boundary, + * so it may need 2. */ -#define SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED | PTE_UXN) -#define SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S | PTE_UXN) - -#ifdef CONFIG_ARM64_4K_PAGES -#define SWAPPER_RW_MMUFLAGS (PMD_ATTRINDX(MT_NORMAL) | SWAPPER_PMD_FLAGS | PTE_WRITE) -#define SWAPPER_RX_MMUFLAGS (SWAPPER_RW_MMUFLAGS | PMD_SECT_RDONLY) +#define EARLY_IDMAP_EXTRA_PAGES 3 +#define EARLY_IDMAP_EXTRA_FDT_PAGES 2 #else -#define SWAPPER_RW_MMUFLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS | PTE_WRITE) -#define SWAPPER_RX_MMUFLAGS (SWAPPER_RW_MMUFLAGS | PTE_RDONLY) +#define EARLY_SEGMENT_EXTRA_PAGES 0 +#define EARLY_IDMAP_EXTRA_PAGES 0 +#define EARLY_IDMAP_EXTRA_FDT_PAGES 0 #endif #endif /* __ASM_KERNEL_PGTABLE_H */ diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index a6fb325424e7..bea13dc4469f 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -61,11 +61,9 @@ static inline void cpu_switch_mm(pgd_t *pgd, struct mm_struct *mm) } /* - * TCR.T0SZ value to use when the ID map is active. Usually equals - * TCR_T0SZ(VA_BITS), unless system RAM is positioned very high in - * physical memory, in which case it will be smaller. + * TCR.T0SZ value to use when the ID map is active. */ -extern int idmap_t0sz; +#define idmap_t0sz TCR_T0SZ(IDMAP_VA_BITS) /* * Ensure TCR.T0SZ is set to the provided value. diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index a1c29d64e875..545b5d8976f4 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -80,26 +80,42 @@ * x19 primary_entry() .. start_kernel() whether we entered with the MMU on * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 - * x22 create_idmap() .. start_kernel() ID map VA of the DT blob * x25 primary_entry() .. start_kernel() supported VA size - * x28 create_idmap() callee preserved temp register */ SYM_CODE_START(primary_entry) bl record_mmu_state bl preserve_boot_args - bl create_idmap + + adrp x1, early_init_stack + mov sp, x1 + mov x29, xzr + adrp x0, init_idmap_pg_dir + bl __pi_create_init_idmap + + /* + * If the page tables have been populated with non-cacheable + * accesses (MMU disabled), invalidate those tables again to + * remove any speculatively loaded cache lines. + */ + cbnz x19, 0f + dmb sy + mov x1, x0 // end of used region + adrp x0, init_idmap_pg_dir + adr_l x2, dcache_inval_poc + blr x2 + b 1f /* * If we entered with the MMU and caches on, clean the ID mapped part * of the primary boot code to the PoC so we can safely execute it with * the MMU off. */ - cbz x19, 0f - adrp x0, __idmap_text_start +0: adrp x0, __idmap_text_start adr_l x1, __idmap_text_end adr_l x2, dcache_clean_poc blr x2 -0: mov x0, x19 + +1: mov x0, x19 bl init_kernel_el // w0=cpu_boot_mode mov x20, x0 @@ -175,238 +191,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) ret SYM_CODE_END(preserve_boot_args) -/* - * Macro to populate page table entries, these entries can be pointers to the next level - * or last level entries pointing to physical memory. - * - * tbl: page table address - * rtbl: pointer to page table or physical memory - * index: start index to write - * eindex: end index to write - [index, eindex] written to - * flags: flags for pagetable entry to or in - * inc: increment to rtbl between each entry - * tmp1: temporary variable - * - * Preserves: tbl, eindex, flags, inc - * Corrupts: index, tmp1 - * Returns: rtbl - */ - .macro populate_entries, tbl, rtbl, index, eindex, flags, inc, tmp1 -.Lpe\@: phys_to_pte \tmp1, \rtbl - orr \tmp1, \tmp1, \flags // tmp1 = table entry - str \tmp1, [\tbl, \index, lsl #3] - add \rtbl, \rtbl, \inc // rtbl = pa next level - add \index, \index, #1 - cmp \index, \eindex - b.ls .Lpe\@ - .endm - -/* - * Compute indices of table entries from virtual address range. If multiple entries - * were needed in the previous page table level then the next page table level is assumed - * to be composed of multiple pages. (This effectively scales the end index). - * - * vstart: virtual address of start of range - * vend: virtual address of end of range - we map [vstart, vend] - * shift: shift used to transform virtual address into index - * order: #imm 2log(number of entries in page table) - * istart: index in table corresponding to vstart - * iend: index in table corresponding to vend - * count: On entry: how many extra entries were required in previous level, scales - * our end index. - * On exit: returns how many extra entries required for next page table level - * - * Preserves: vstart, vend - * Returns: istart, iend, count - */ - .macro compute_indices, vstart, vend, shift, order, istart, iend, count - ubfx \istart, \vstart, \shift, \order - ubfx \iend, \vend, \shift, \order - add \iend, \iend, \count, lsl \order - sub \count, \iend, \istart - .endm - -/* - * Map memory for specified virtual address range. Each level of page table needed supports - * multiple entries. If a level requires n entries the next page table level is assumed to be - * formed from n pages. - * - * tbl: location of page table - * rtbl: address to be used for first level page table entry (typically tbl + PAGE_SIZE) - * vstart: virtual address of start of range - * vend: virtual address of end of range - we map [vstart, vend - 1] - * flags: flags to use to map last level entries - * phys: physical address corresponding to vstart - physical memory is contiguous - * order: #imm 2log(number of entries in PGD table) - * - * If extra_shift is set, an extra level will be populated if the end address does - * not fit in 'extra_shift' bits. This assumes vend is in the TTBR0 range. - * - * Temporaries: istart, iend, tmp, count, sv - these need to be different registers - * Preserves: vstart, flags - * Corrupts: tbl, rtbl, vend, istart, iend, tmp, count, sv - */ - .macro map_memory, tbl, rtbl, vstart, vend, flags, phys, order, istart, iend, tmp, count, sv, extra_shift - sub \vend, \vend, #1 - add \rtbl, \tbl, #PAGE_SIZE - mov \count, #0 - - .ifnb \extra_shift - tst \vend, #~((1 << (\extra_shift)) - 1) - b.eq .L_\@ - compute_indices \vstart, \vend, #\extra_shift, #(PAGE_SHIFT - 3), \istart, \iend, \count - mov \sv, \rtbl - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - .endif -.L_\@: - compute_indices \vstart, \vend, #PGDIR_SHIFT, #\order, \istart, \iend, \count - mov \sv, \rtbl - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - -#if SWAPPER_PGTABLE_LEVELS > 3 - compute_indices \vstart, \vend, #PUD_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count - mov \sv, \rtbl - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv -#endif - -#if SWAPPER_PGTABLE_LEVELS > 2 - compute_indices \vstart, \vend, #SWAPPER_TABLE_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count - mov \sv, \rtbl - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv -#endif - - compute_indices \vstart, \vend, #SWAPPER_BLOCK_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count - bic \rtbl, \phys, #SWAPPER_BLOCK_SIZE - 1 - populate_entries \tbl, \rtbl, \istart, \iend, \flags, #SWAPPER_BLOCK_SIZE, \tmp - .endm - -/* - * Remap a subregion created with the map_memory macro with modified attributes - * or output address. The entire remapped region must have been covered in the - * invocation of map_memory. - * - * x0: last level table address (returned in first argument to map_memory) - * x1: start VA of the existing mapping - * x2: start VA of the region to update - * x3: end VA of the region to update (exclusive) - * x4: start PA associated with the region to update - * x5: attributes to set on the updated region - * x6: order of the last level mappings - */ -SYM_FUNC_START_LOCAL(remap_region) - sub x3, x3, #1 // make end inclusive - - // Get the index offset for the start of the last level table - lsr x1, x1, x6 - bfi x1, xzr, #0, #PAGE_SHIFT - 3 - - // Derive the start and end indexes into the last level table - // associated with the provided region - lsr x2, x2, x6 - lsr x3, x3, x6 - sub x2, x2, x1 - sub x3, x3, x1 - - mov x1, #1 - lsl x6, x1, x6 // block size at this level - - populate_entries x0, x4, x2, x3, x5, x6, x7 - ret -SYM_FUNC_END(remap_region) - -SYM_FUNC_START_LOCAL(create_idmap) - mov x28, lr - /* - * The ID map carries a 1:1 mapping of the physical address range - * covered by the loaded image, which could be anywhere in DRAM. This - * means that the required size of the VA (== PA) space is decided at - * boot time, and could be more than the configured size of the VA - * space for ordinary kernel and user space mappings. - * - * There are three cases to consider here: - * - 39 <= VA_BITS < 48, and the ID map needs up to 48 VA bits to cover - * the placement of the image. In this case, we configure one extra - * level of translation on the fly for the ID map only. (This case - * also covers 42-bit VA/52-bit PA on 64k pages). - * - * - VA_BITS == 48, and the ID map needs more than 48 VA bits. This can - * only happen when using 64k pages, in which case we need to extend - * the root level table rather than add a level. Note that we can - * treat this case as 'always extended' as long as we take care not - * to program an unsupported T0SZ value into the TCR register. - * - * - Combinations that would require two additional levels of - * translation are not supported, e.g., VA_BITS==36 on 16k pages, or - * VA_BITS==39/4k pages with 5-level paging, where the input address - * requires more than 47 or 48 bits, respectively. - */ -#if (VA_BITS < 48) -#define IDMAP_PGD_ORDER (VA_BITS - PGDIR_SHIFT) -#define EXTRA_SHIFT (PGDIR_SHIFT + PAGE_SHIFT - 3) - - /* - * If VA_BITS < 48, we have to configure an additional table level. - * First, we have to verify our assumption that the current value of - * VA_BITS was chosen such that all translation levels are fully - * utilised, and that lowering T0SZ will always result in an additional - * translation level to be configured. - */ -#if VA_BITS != EXTRA_SHIFT -#error "Mismatch between VA_BITS and page size/number of translation levels" -#endif -#else -#define IDMAP_PGD_ORDER (PHYS_MASK_SHIFT - PGDIR_SHIFT) -#define EXTRA_SHIFT - /* - * If VA_BITS == 48, we don't have to configure an additional - * translation level, but the top-level table has more entries. - */ -#endif - adrp x0, init_idmap_pg_dir - adrp x3, _text - adrp x6, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE - mov_q x7, SWAPPER_RX_MMUFLAGS - - map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT - - /* Remap [.init].data, BSS and the kernel page tables r/w in the ID map */ - adrp x1, _text - adrp x2, __initdata_begin - adrp x3, _end - bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 - mov_q x5, SWAPPER_RW_MMUFLAGS - mov x6, #SWAPPER_BLOCK_SHIFT - bl remap_region - - /* Remap the FDT after the kernel image */ - adrp x1, _text - adrp x22, _end + SWAPPER_BLOCK_SIZE - bic x2, x22, #SWAPPER_BLOCK_SIZE - 1 - bfi x22, x21, #0, #SWAPPER_BLOCK_SHIFT // remapped FDT address - add x3, x2, #MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE - bic x4, x21, #SWAPPER_BLOCK_SIZE - 1 - mov_q x5, SWAPPER_RW_MMUFLAGS - mov x6, #SWAPPER_BLOCK_SHIFT - bl remap_region - - /* - * Since the page tables have been populated with non-cacheable - * accesses (MMU disabled), invalidate those tables again to - * remove any speculatively loaded cache lines. - */ - cbnz x19, 0f // skip cache invalidation if MMU is on - dmb sy - - adrp x0, init_idmap_pg_dir - adrp x1, init_idmap_pg_end - bl dcache_inval_poc -0: ret x28 -SYM_FUNC_END(create_idmap) - /* * Initialize CPU registers with task-specific and cpu-specific context. * @@ -729,11 +513,6 @@ SYM_FUNC_START_LOCAL(__no_granule_support) SYM_FUNC_END(__no_granule_support) SYM_FUNC_START_LOCAL(__primary_switch) - mrs x1, tcr_el1 - mov x2, #64 - VA_BITS - tcr_set_t0sz x1, x2 - msr tcr_el1, x1 - adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu @@ -742,7 +521,7 @@ SYM_FUNC_START_LOCAL(__primary_switch) mov sp, x1 mov x29, xzr mov x0, x20 // pass the full boot status - mov x1, x22 // pass the low FDT mapping + mov x1, x21 // pass the FDT bl __pi_early_map_kernel // Map and relocate the kernel ldr x8, =__primary_switched diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index bb108f4f9b75..cec76aa8efb9 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -50,6 +50,7 @@ PROVIDE(__pi_cavium_erratum_27456_cpus = cavium_erratum_27456_cpus); PROVIDE(__pi__ctype = _ctype); PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); +PROVIDE(__pi_init_idmap_pg_dir = init_idmap_pg_dir); PROVIDE(__pi_init_pg_dir = init_pg_dir); PROVIDE(__pi_init_pg_end = init_pg_end); diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 8c2f80a46b93..4393b41f0b71 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -11,6 +11,9 @@ KBUILD_CFLAGS := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) -fpie \ -fno-asynchronous-unwind-tables -fno-unwind-tables \ $(call cc-option,-fno-addrsig) +# this code may run with the MMU off so disable unaligned accesses +CFLAGS_map_range.o += -mstrict-align + # remove SCS flags from all objects in this directory KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS)) # disable LTO diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index 9e64c14c1598..e7edf5e9f457 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -129,6 +129,22 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) } } +static void map_fdt(u64 fdt) +{ + static u8 ptes[INIT_IDMAP_FDT_SIZE] __initdata __aligned(PAGE_SIZE); + u64 efdt = fdt + MAX_FDT_SIZE; + u64 ptep = (u64)ptes; + + /* + * Map up to MAX_FDT_SIZE bytes, but avoid overlap with + * the kernel image. + */ + map_range(&ptep, fdt, (u64)_text > fdt ? min((u64)_text, efdt) : efdt, + fdt, PAGE_KERNEL, IDMAP_ROOT_LEVEL, + (pte_t *)init_idmap_pg_dir, false, 0); + dsb(ishst); +} + asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) { static char const chosen_str[] __initconst = "/chosen"; @@ -137,6 +153,8 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) int root_level = 4 - CONFIG_PGTABLE_LEVELS; int chosen; + map_fdt((u64)fdt); + /* Clear BSS and the initial page tables */ memset(__bss_start, 0, (u64)init_pg_end - (u64)__bss_start); diff --git a/arch/arm64/kernel/pi/map_range.c b/arch/arm64/kernel/pi/map_range.c index c31feda18f47..79e4f6a2efe1 100644 --- a/arch/arm64/kernel/pi/map_range.c +++ b/arch/arm64/kernel/pi/map_range.c @@ -86,3 +86,15 @@ void __init map_range(u64 *pte, u64 start, u64 end, u64 pa, pgprot_t prot, tbl++; } } + +asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir) +{ + u64 ptep = (u64)pg_dir + PAGE_SIZE; + + map_range(&ptep, (u64)_stext, (u64)__initdata_begin, (u64)_stext, + PAGE_KERNEL_ROX, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); + map_range(&ptep, (u64)__initdata_begin, (u64)_end, (u64)__initdata_begin, + PAGE_KERNEL, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); + + return ptep; +} diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h index dfc70828ad0a..a8fe79c9b111 100644 --- a/arch/arm64/kernel/pi/pi.h +++ b/arch/arm64/kernel/pi/pi.h @@ -17,6 +17,8 @@ static inline void *prel64_to_pointer(const prel64_t *offset) extern bool dynamic_scs_is_enabled; +extern pgd_t init_idmap_pg_dir[]; + void init_feature_override(u64 boot_status, const void *fdt, int chosen); u64 kaslr_early_init(void *fdt, int chosen); void relocate_kernel(u64 offset); @@ -24,3 +26,5 @@ int scs_patch(const u8 eh_frame[], int size); void map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, int level, pte_t *tbl, bool may_use_cont, u64 va_offset); + +asmlinkage u64 create_init_idmap(pgd_t *pgd); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 070bc1bc5ad1..9d965c4df65d 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -45,8 +45,6 @@ #define NO_CONT_MAPPINGS BIT(1) #define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ -int idmap_t0sz __ro_after_init; - #if VA_BITS > 48 u64 vabits_actual __ro_after_init = VA_BITS_MIN; EXPORT_SYMBOL(vabits_actual); @@ -790,8 +788,6 @@ void __init paging_init(void) pgd_t *pgdp = pgd_set_fixmap(__pa_symbol(swapper_pg_dir)); extern pgd_t init_idmap_pg_dir[]; - idmap_t0sz = 63UL - __fls(__pa_symbol(_end) | GENMASK(VA_BITS_MIN - 1, 0)); - map_kernel(pgdp); map_mem(pgdp); @@ -806,7 +802,6 @@ void __init paging_init(void) memblock_allow_resize(); create_idmap(); - idmap_t0sz = TCR_T0SZ(IDMAP_VA_BITS); } #ifdef CONFIG_MEMORY_HOTPLUG diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 8432af63e023..6d2de0996c00 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -200,7 +200,8 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 -#define KPTI_NG_PTE_FLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS | PTE_WRITE) +#define KPTI_NG_PTE_FLAGS (PTE_ATTRINDX(MT_NORMAL) | PTE_TYPE_PAGE | \ + PTE_AF | PTE_SHARED | PTE_UXN | PTE_WRITE) .pushsection ".idmap.text", "a" From patchwork Tue Sep 12 14:16:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C80F2CA0EEC for ; Tue, 12 Sep 2023 14:20:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=EsYhTljosz0+LCNuhjqzFoPMXIJNI+ZjuqZzaDBNbUM=; b=1sGqn7kzaAGvuaWgS4klHGJCB+ J5HD4SgUJgKmdPneL3v5Buk26qIkHE5BGLTXOKP0PS6XJFJ5aTzfOhMirhQj9DujrPz1bBuwf/qHN 4PST9DO03xX5cHktLczG2c2a1wCVbEDM7tM7UHCNvSqowtqJdALAFJgqHWXu7VmExvBy5FiR2TeRv lD/G2b+FnaVckP1RVzexQ9la/y8IGSS24ethr9AQIgvnAq6p3Qu1xKKh5WE6bMxkpnyMG64d6rSB7 Sb7GcqltVzK/ZEuYA2z00qhVhOVVy70pBYMDVg7O9tuAHgZ9oQhL91OX0ko2GjdqFw+j2KXBslYnu Dv0AGmFw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4F8-003Xqn-1e; Tue, 12 Sep 2023 14:19:54 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4E4-003WvP-2Z for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:03 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-30932d15a30so4056232f8f.1 for ; Tue, 12 Sep 2023 07:18:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528326; x=1695133126; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lu+998G42yoYzQEKfrWeyih/jY59r+JYoyBx68HGSFI=; b=gPxOYY+fN0UwnB4wb/iop5QxZ4AnM0BIt21P1FocunIgatNI0e/gcCMa3lFEvz+6vG JyHERBKwTwJsCMfs1ylDkRVOQ6sbh6IVaCPZ8puDVs4kpLqWlJVNfSVQil2IlH4EejTK ydLrsyC2vi9xiTGeloi9AGG+uS44xo9RG/xbLupGxCkryNy/L9L+GPx+ftHOYPVGgFaJ NmPC874EUbRydqSp3kTcXes2G2wuJIMHf8zw4PvuWqrVyjTQQrNLGYounKO3PxU946wE bUbTYpN4CzQWVxh5wvmPxYDbRjgbHfylAvmVd6mACiw+o+nxx8GU+yLMn36YLFDDnjhI 1/Lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528326; x=1695133126; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lu+998G42yoYzQEKfrWeyih/jY59r+JYoyBx68HGSFI=; b=b+5rvWrPtmHWtFtTx8qpYPr27TWNv4figLqnW1hmUDSlFAgBNL1D+N2y027TSmiGmF gxrzyXhJVptIqH5J0EIqthAz/JWmySy8tuJIv/jJ6T+wWkm74ti9qxXc3UOCgRDdnJuz mDbnGUAdxCxQrQJhzlZQex3Wn3a7LaTCTH8epyhf0YHty8B6QrBwd+uIIYT+Q44mht85 yQjwHPCaI7q8Rve2Mk8tRWM2O8x7hIU4+ulBaRP5hGl8J9SFgklzCGJHbfk4KGJy4SNk fGn6XAMXuBSzYXQQ4bIzaIpVxfDPnlJ1bcXQWRGeHEIX21yWbM7QQskzMySXvIBCPAyv cEnA== X-Gm-Message-State: AOJu0Yy+VOi2BoS67g+PRNSu4MM1RWnORs37wFTctMY9aiP3wuoWQKhx ACk2nJeJ6x7KZ2sRuqv+wuAaGMkD3BVVaKDGtsuc+336XH1oEqj3eVtyb/uEfZYEPBLmTpb6KHv nJLHCPUYn4uvi0GRDEzXxyp5Zpn0mA6uZoHC6MHZYv1VEg/688VctiWconhx5nySRxtUD29slM5 s= X-Google-Smtp-Source: AGHT+IHa/pWhDddtcvlLtch8bevNYQuYRuskg5kcTws4jb1bcdKCg8dFqCBDt4bemPwUfCWsdjfqwqTU X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:d1e4:0:b0:31f:9838:dfbc with SMTP id g4-20020adfd1e4000000b0031f9838dfbcmr107432wrd.11.1694528325803; Tue, 12 Sep 2023 07:18:45 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:27 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2122; i=ardb@kernel.org; h=from:subject; bh=o9DIE+pNddG54HtQ0IrOHmtMLIY+G8vnueC1wug+zzo=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaM/EB6s4TpzfVaOd5bvk7vpXmeKvHl/eqLOhJLeLe 0OxEK93RykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjIyX+MDO+unTpU8napMktV a3/+u3IpyU9JEt+9HeecywqM6Lpy8QnDf2/lmJnftKomHJuwcZXr4e1OR8w/frym5DLnvCXndLu 6fH4A X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-100-ardb@google.com> Subject: [PATCH v4 37/61] arm64: mm: avoid fixmap for early swapper_pg_dir updates From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071848_891167_59D0D2B4 X-CRM114-Status: GOOD ( 15.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Early in the boot, when .rodata is still writable, we can poke swapper_pg_dir entries directly, and there is no need to go through the fixmap. After a future patch, we will enter the kernel with swapper_pg_dir already active, and early swapper_pg_dir updates for creating the fixmap page table hierarchy itself cannot go through the fixmap for obvious reaons. So let's keep track of whether rodata is writable, and update the descriptor directly in that case. As the same reasoning applies to early KASAN init, make the function noinstr as well. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/mmu.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 9d965c4df65d..ae2134be6be5 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -55,6 +55,8 @@ EXPORT_SYMBOL(kimage_voffset); u32 __boot_cpu_mode[] = { BOOT_CPU_MODE_EL2, BOOT_CPU_MODE_EL1 }; +static bool rodata_is_rw __ro_after_init = true; + /* * The booting CPU updates the failed status @__early_cpu_boot_status, * with MMU turned off. @@ -71,10 +73,21 @@ EXPORT_SYMBOL(empty_zero_page); static DEFINE_SPINLOCK(swapper_pgdir_lock); static DEFINE_MUTEX(fixmap_lock); -void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) +void noinstr set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) { pgd_t *fixmap_pgdp; + /* + * Don't bother with the fixmap if swapper_pg_dir is still mapped + * writable in the kernel mapping. + */ + if (rodata_is_rw) { + WRITE_ONCE(*pgdp, pgd); + dsb(ishst); + isb(); + return; + } + spin_lock(&swapper_pgdir_lock); fixmap_pgdp = pgd_set_fixmap(__pa_symbol(pgdp)); WRITE_ONCE(*fixmap_pgdp, pgd); @@ -628,6 +641,7 @@ void mark_rodata_ro(void) * to cover NOTES and EXCEPTION_TABLE. */ section_size = (unsigned long)__init_begin - (unsigned long)__start_rodata; + WRITE_ONCE(rodata_is_rw, false); update_mapping_prot(__pa_symbol(__start_rodata), (unsigned long)__start_rodata, section_size, PAGE_KERNEL_RO); From patchwork Tue Sep 12 14:16:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381797 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E2AD2CA0EEC for ; Tue, 12 Sep 2023 14:20:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=XV4HwkK+4A3H310ow03n9jttp446/9t3IjTHuR1JAjU=; b=m56iAkiWAys2n7K7R9JIAPT+2l rR1O4rCzbTVqIyvkkNNNJPbvfUrT65gF7sOPI5EJnhnMdz1YDR6HHZJtysLl75TbrAWfG/DoQevTk Gy6CwKJQMTJLW/K0bbuPxdUHoa/ysa++aCeaNEU+AUc7I1G9a+TWa4mL2srQFechlLG02dSaBj5wQ 99P9IaIigc6/fX+eg70jg4kjSUo53o2F32VBae0hSa3xtJy4N1KlqNudfro84HKKr4qe/OslQtPp1 TUuJsuugTsNuxiHJzeN3OpTv4q0IpGKW++dPn6v7EA+C3XOtTNvXR/BXDHUsygZYK37H62tI7adO2 ld+jop8g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Eq-003XcV-30; Tue, 12 Sep 2023 14:19:36 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4EB-003X21-1i for linux-arm-kernel@bombadil.infradead.org; Tue, 12 Sep 2023 14:18:55 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=/aKYFJPr9sy1R/AIhdxam28biyUtBu1aC+8KzdeY6EY=; b=kEsxCrm6ZutS5JQEohwlYfzdbA NpV/QwRcrAdgxy3DDNGFYR6NQB4ErURxvlxqqSe5qOSgdvN/nmZk6LjHdWGz86Duyj/WoPlafMgjc Ohjd/NPzO8asDiN4BucvywnG45z6/8uEgn4Ir9Ct+hsDporfIVV99ZF1/KjBL98r3roC0abxWGQJu caivlbY5tJWMnzq00gDFM0h3yisYzaMlTnuRCR8uoPJFslyoluEkoPwMzLfsnOdgHHHmYpEb3mSqE tGV5x3o+9EZvlI4khKcYf0H2xL+pm7e3ECGp8Kk4/rYyWXEgXRQtmfUTB83x51A1UphqAWHhWW28r q8J/9yrw==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4E5-006A1C-1z for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:18:54 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-59224c40275so62559437b3.3 for ; Tue, 12 Sep 2023 07:18:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528328; x=1695133128; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/aKYFJPr9sy1R/AIhdxam28biyUtBu1aC+8KzdeY6EY=; b=vNNdCPOoc0iBjGzUWRkUjCnx6n0to8MuTHr5z4luw/OPWzpdumuByu50PY3EsAvkDK t/1564unZ18rx6a96hDQurARi6etX1GYLr4GZ1MHdfarcuGKj81MGnvJ8v3FSoBg6pgo zmUDVz9hQuLSTZ4gTIygjoksW28+tmfjtDEFkB+DNHL2b0JYGAbQKzvcqt1XszWitMFp kX7hPdsNQ92q+eHnSsvxkIlIz1zVN2pzruhijU2nypamrohJp6nA58MAd5KCbQzUjnNs RINLVHmBakdnBmCfd9MpQ4yQEjV+SFQnesFipkz3YekvBBBYpAfinIIHCxlf/N2ybIcX jytw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528328; x=1695133128; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/aKYFJPr9sy1R/AIhdxam28biyUtBu1aC+8KzdeY6EY=; b=FRNwNZgVVkzr9SG4iAXNPfcSTqMfs5lxuxoQRtDL8GKAFBCHS6+cTAZGvqXOQ4mG/D SO5DtDRZ303Q0AOKBFg69iEGRDXgWNKjVGOfGG7n0tEFHsqeEkL+Ct0BkvRWkfdZn+Hc T87X/NWzC8H1wogcwgmL9ydwsApGGmeDCZzgE0B4v0uQWmjLqU4SkcruJN1/LeE6uvZl QeFkkixL1MR/oMD7VOUALu28YvGDGZl163eW8GSlxgG19F6mFadzc/QnWYEl7EoLJp/X CdgjIZMeqQCWZUjAi8yvO4CBvYhNBQz77/cM2pV0EbnS2Puh6PfM8lM08vSGYQKvogNc c5SA== X-Gm-Message-State: AOJu0Yx9qZ35c4CR5Y9vm7FJed8ffTuw7s+GirDN9VGPxwtggmExOEpv GM/1YfcyGYpD242OCWpCZvX0SJXb04Z0vsjGvOAiip7sQIGbF2qD6NEgda+wLKGBIDZfZRJFim8 gPfGoh210b1SVJR4ODCA4Hcnjziv9pDfLpt+DJ1HRVbD2cfHw06yDaz1yDFermkTzEfwavFznKE o= X-Google-Smtp-Source: AGHT+IEzGzP/oTp6TNM5zLM/LYR+tWy9hG1ncSh60Di17Xej1zlylmN7r1VOrW94i79rj/xJZPxlLx00 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:c84a:0:b0:584:3d8f:a423 with SMTP id k10-20020a81c84a000000b005843d8fa423mr299910ywl.8.1694528328446; Tue, 12 Sep 2023 07:18:48 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:28 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10396; i=ardb@kernel.org; h=from:subject; bh=uHQLYijr4Y8jm92LE4HEMZzVB3WvTVfwzO+S3yannyo=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaF/7JgsRVqWE9WqTjk/OzTryyezRqRl3Oe78DCuv3 SQlXKjXUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACZyeBbDb5bj/fv0zuo6vOeX vvxf6XFCQh5ru/VdXcbQxTJeOyrqxBn+aYSVsT//qFXp8KPgWO7tj5map3iVohI5HHdPLtF5fSy IBwA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-101-ardb@google.com> Subject: [PATCH v4 38/61] arm64: mm: omit redundant remap of kernel image From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_151851_244314_620795C1 X-CRM114-Status: GOOD ( 27.61 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that the early kernel mapping is created with all the right attributes and segment boundaries, there is no longer a need to recreate it and switch to it. This also means we no longer have to copy the kasan shadow or some parts of the fixmap from one set of page tables to the other. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/fixmap.h | 1 - arch/arm64/include/asm/kasan.h | 2 - arch/arm64/include/asm/mmu.h | 2 +- arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/map_kernel.c | 6 +- arch/arm64/mm/fixmap.c | 34 -------- arch/arm64/mm/kasan_init.c | 15 ---- arch/arm64/mm/mmu.c | 85 ++++---------------- 8 files changed, 21 insertions(+), 125 deletions(-) diff --git a/arch/arm64/include/asm/fixmap.h b/arch/arm64/include/asm/fixmap.h index 58c294a96676..8aabd45e9a13 100644 --- a/arch/arm64/include/asm/fixmap.h +++ b/arch/arm64/include/asm/fixmap.h @@ -100,7 +100,6 @@ enum fixed_addresses { #define FIXMAP_PAGE_IO __pgprot(PROT_DEVICE_nGnRE) void __init early_fixmap_init(void); -void __init fixmap_copy(pgd_t *pgdir); #define __early_set_fixmap __set_fixmap diff --git a/arch/arm64/include/asm/kasan.h b/arch/arm64/include/asm/kasan.h index 12d5f47f7dbe..ab52688ac4bd 100644 --- a/arch/arm64/include/asm/kasan.h +++ b/arch/arm64/include/asm/kasan.h @@ -36,12 +36,10 @@ void kasan_init(void); #define _KASAN_SHADOW_START(va) (KASAN_SHADOW_END - (1UL << ((va) - KASAN_SHADOW_SCALE_SHIFT))) #define KASAN_SHADOW_START _KASAN_SHADOW_START(vabits_actual) -void kasan_copy_shadow(pgd_t *pgdir); asmlinkage void kasan_early_init(void); #else static inline void kasan_init(void) { } -static inline void kasan_copy_shadow(pgd_t *pgdir) { } #endif #endif diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index be41054a500e..8eec57bf74c2 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -110,7 +110,7 @@ static inline bool kaslr_requires_kpti(void) } #define INIT_MM_CONTEXT(name) \ - .pgd = init_pg_dir, + .pgd = swapper_pg_dir, #endif /* !__ASSEMBLY__ */ #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index cec76aa8efb9..2ae33cc02051 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -53,6 +53,7 @@ PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); PROVIDE(__pi_init_idmap_pg_dir = init_idmap_pg_dir); PROVIDE(__pi_init_pg_dir = init_pg_dir); PROVIDE(__pi_init_pg_end = init_pg_end); +PROVIDE(__pi_swapper_pg_dir = swapper_pg_dir); PROVIDE(__pi__text = _text); PROVIDE(__pi__stext = _stext); diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index e7edf5e9f457..d0fad93a6872 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -125,8 +125,12 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) text_prot, true, root_level); map_segment(init_pg_dir, NULL, va_offset, __inittext_begin, __inittext_end, text_prot, false, root_level); - dsb(ishst); } + + /* Copy the root page table to its final location */ + memcpy((void *)swapper_pg_dir + va_offset, init_pg_dir, PGD_SIZE); + dsb(ishst); + idmap_cpu_replace_ttbr1(swapper_pg_dir); } static void map_fdt(u64 fdt) diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index c0a3301203bd..9436a12e1882 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -167,37 +167,3 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) return dt_virt; } - -/* - * Copy the fixmap region into a new pgdir. - */ -void __init fixmap_copy(pgd_t *pgdir) -{ - if (!READ_ONCE(pgd_val(*pgd_offset_pgd(pgdir, FIXADDR_TOT_START)))) { - /* - * The fixmap falls in a separate pgd to the kernel, and doesn't - * live in the carveout for the swapper_pg_dir. We can simply - * re-use the existing dir for the fixmap. - */ - set_pgd(pgd_offset_pgd(pgdir, FIXADDR_TOT_START), - READ_ONCE(*pgd_offset_k(FIXADDR_TOT_START))); - } else if (CONFIG_PGTABLE_LEVELS > 3) { - pgd_t *bm_pgdp; - p4d_t *bm_p4dp; - pud_t *bm_pudp; - /* - * The fixmap shares its top level pgd entry with the kernel - * mapping. This can really only occur when we are running - * with 16k/4 levels, so we can simply reuse the pud level - * entry instead. - */ - BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES)); - bm_pgdp = pgd_offset_pgd(pgdir, FIXADDR_TOT_START); - bm_p4dp = p4d_offset(bm_pgdp, FIXADDR_TOT_START); - bm_pudp = pud_set_fixmap_offset(bm_p4dp, FIXADDR_TOT_START); - pud_populate(&init_mm, bm_pudp, lm_alias(bm_pmd)); - pud_clear_fixmap(); - } else { - BUG(); - } -} diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index f17d066e85eb..856bda5fa19d 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -184,21 +184,6 @@ static void __init kasan_map_populate(unsigned long start, unsigned long end, kasan_pgd_populate(start & PAGE_MASK, PAGE_ALIGN(end), node, false); } -/* - * Copy the current shadow region into a new pgdir. - */ -void __init kasan_copy_shadow(pgd_t *pgdir) -{ - pgd_t *pgdp, *pgdp_new, *pgdp_end; - - pgdp = pgd_offset_k(KASAN_SHADOW_START); - pgdp_end = pgd_offset_k(KASAN_SHADOW_END); - pgdp_new = pgd_offset_pgd(pgdir, KASAN_SHADOW_START); - do { - set_pgd(pgdp_new, READ_ONCE(*pgdp)); - } while (pgdp++, pgdp_new++, pgdp != pgdp_end); -} - static void __init clear_pgds(unsigned long start, unsigned long end) { diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index ae2134be6be5..1f74faffa408 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -648,9 +648,9 @@ void mark_rodata_ro(void) debug_checkwx(); } -static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, - pgprot_t prot, struct vm_struct *vma, - int flags, unsigned long vm_flags) +static void __init declare_vma(struct vm_struct *vma, + void *va_start, void *va_end, + unsigned long vm_flags) { phys_addr_t pa_start = __pa_symbol(va_start); unsigned long size = va_end - va_start; @@ -658,9 +658,6 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, BUG_ON(!PAGE_ALIGNED(pa_start)); BUG_ON(!PAGE_ALIGNED(size)); - __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, prot, - early_pgtable_alloc, flags); - if (!(vm_flags & VM_NO_GUARD)) size += PAGE_SIZE; @@ -673,12 +670,12 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, vm_area_add_early(vma); } +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 static pgprot_t kernel_exec_prot(void) { return rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC; } -#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 static int __init map_entry_trampoline(void) { int i; @@ -710,60 +707,17 @@ core_initcall(map_entry_trampoline); #endif /* - * Open coded check for BTI, only for use to determine configuration - * for early mappings for before the cpufeature code has run. - */ -static bool arm64_early_this_cpu_has_bti(void) -{ - u64 pfr1; - - if (!IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) - return false; - - pfr1 = __read_sysreg_by_encoding(SYS_ID_AA64PFR1_EL1); - return cpuid_feature_extract_unsigned_field(pfr1, - ID_AA64PFR1_EL1_BT_SHIFT); -} - -/* - * Create fine-grained mappings for the kernel. + * Declare the VMA areas for the kernel */ -static void __init map_kernel(pgd_t *pgdp) +static void __init declare_kernel_vmas(void) { - static struct vm_struct vmlinux_text, vmlinux_rodata, vmlinux_inittext, - vmlinux_initdata, vmlinux_data; - - /* - * External debuggers may need to write directly to the text - * mapping to install SW breakpoints. Allow this (only) when - * explicitly requested with rodata=off. - */ - pgprot_t text_prot = kernel_exec_prot(); - - /* - * If we have a CPU that supports BTI and a kernel built for - * BTI then mark the kernel executable text as guarded pages - * now so we don't have to rewrite the page tables later. - */ - if (arm64_early_this_cpu_has_bti()) - text_prot = __pgprot_modify(text_prot, PTE_GP, PTE_GP); + static struct vm_struct vmlinux_seg[KERNEL_SEGMENT_COUNT]; - /* - * Only rodata will be remapped with different permissions later on, - * all other segments are allowed to use contiguous mappings. - */ - map_kernel_segment(pgdp, _stext, _etext, text_prot, &vmlinux_text, 0, - VM_NO_GUARD); - map_kernel_segment(pgdp, __start_rodata, __inittext_begin, PAGE_KERNEL, - &vmlinux_rodata, NO_CONT_MAPPINGS, VM_NO_GUARD); - map_kernel_segment(pgdp, __inittext_begin, __inittext_end, text_prot, - &vmlinux_inittext, 0, VM_NO_GUARD); - map_kernel_segment(pgdp, __initdata_begin, __initdata_end, PAGE_KERNEL, - &vmlinux_initdata, 0, VM_NO_GUARD); - map_kernel_segment(pgdp, _data, _end, PAGE_KERNEL, &vmlinux_data, 0, 0); - - fixmap_copy(pgdp); - kasan_copy_shadow(pgdp); + declare_vma(&vmlinux_seg[0], _stext, _etext, VM_NO_GUARD); + declare_vma(&vmlinux_seg[1], __start_rodata, __inittext_begin, VM_NO_GUARD); + declare_vma(&vmlinux_seg[2], __inittext_begin, __inittext_end, VM_NO_GUARD); + declare_vma(&vmlinux_seg[3], __initdata_begin, __initdata_end, VM_NO_GUARD); + declare_vma(&vmlinux_seg[4], _data, _end, 0); } void __pi_map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, @@ -799,23 +753,12 @@ static void __init create_idmap(void) void __init paging_init(void) { - pgd_t *pgdp = pgd_set_fixmap(__pa_symbol(swapper_pg_dir)); - extern pgd_t init_idmap_pg_dir[]; - - map_kernel(pgdp); - map_mem(pgdp); - - pgd_clear_fixmap(); - - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), init_idmap_pg_dir); - init_mm.pgd = swapper_pg_dir; - - memblock_phys_free(__pa_symbol(init_pg_dir), - __pa_symbol(init_pg_end) - __pa_symbol(init_pg_dir)); + map_mem(swapper_pg_dir); memblock_allow_resize(); create_idmap(); + declare_kernel_vmas(); } #ifdef CONFIG_MEMORY_HOTPLUG From patchwork Tue Sep 12 14:16:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 48E63CA0EEB for ; Tue, 12 Sep 2023 14:20:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=T7a0Psm0WaGnjoltSJwokKaaQgS2FwPz68/PXk9kLfY=; b=ju0rW+VHISuxxI5KX1cQk7OO8o rBfuYz+a4LzFNT32mY23sOsmsm60GrmEsDxoavZhMb7lEj76Nutc1X1ccxLU2zgl+s0E82ui0T0PJ ukUFe4VcQZUjBdAmlV6ePJGRc/6IRIpQsxdrVZfuaRkFUQm0AA6r8hqM2ci353wKnNp4v2FyLcUvA XrW5ts47aZPb8Vt/gWsHGNVgY35YoBqVqPcPHJPTZFjhS4uGR/JvR2EpraYIrcrp1Q+3pMZqhI5rr 0sAWCxDSVa3+0TWBqavVjPYeZ+dlJTOJtgR4luNOJCzKA+R0uMim6YuKulIaEu0U4jC9k9xmZmwuK 2ew7/Dww==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4FM-003Y2e-0e; Tue, 12 Sep 2023 14:20:08 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4E8-003Wzk-35 for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:10 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-58d9e327d3aso61945467b3.3 for ; Tue, 12 Sep 2023 07:18:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528331; x=1695133131; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=aLPqD5n9wVyy6M59tP+XXcO5gA8PIr1yOKAAodK2KXs=; b=F6+KzdWGCadlrLlKks4IlJ5GfTOofMwc0rVkjuhJYklyv1O5Bm26PLl/2tRWNfTj43 1gzVQbCk8ZKKnK1e72PYhCfchWY0vPe7EyjIuH7J35ZgLbCapmoMvNsfw9tr1q6wbYcu gJmNXvEy7eryy8jY85sE+xoN+fUkewxqPYx7883Xg9m1Yj4S8xTAvtqjWFfDDdKvWKmQ HMtfAq8vS2/cBgQhIjbypG+yju1Ge777FfUsy2DAjF8Va/+dPpbcO8GPHmkCtIGIX+Yj 2uAvRuxCGkYQwDIfhP1d/+mXOOR71wgmA+oMoMneuorySiT2daJfzwx4Yke95AzAjmiG d28A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528331; x=1695133131; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aLPqD5n9wVyy6M59tP+XXcO5gA8PIr1yOKAAodK2KXs=; b=PiDCMqQw+5MXFIkbKhaWwz3/ZBJFxIoHA+MuikQ0gyynJiB2oAWu3xSCU7UQZdXR1i bdoIc1NEXJD9bXzprJhRkBhi0sSb3wl9XuFVL7VS6ChpF9TkxehuiFVYQEuvM3eRi29H zwaYTzaTRnlpBl1HTMkjgIjIYunEhxrCJveVNMEQVrC4wz+ETjbv8xdjGgvzBxg7JhN4 UwiSbK/4sPypNbgpYQisxOWyuwSkrOimyLg7jTLMlu/igTL6tXg/8Lb3adEo826vXmVP AixFn+uR3eUw5djKBshrUsHRdA6CtacSv0uBicCc6c3WILTblCTABr25vEfoO/0Ssh4F +7wQ== X-Gm-Message-State: AOJu0YzL/VFN9yP+U8WJDswDkDhJ3aYqm92sQfewKLNAcFA6Y25TSDII SzpUrTiYfDxfZY5Gbmb6eFit7GT3my54tbA6lkUyjYgKmN8YQnqLW4ukayOSM0MxXGukXVwRpvL ogII6bnPmDXMG/lHF+WTkvkbPaTe9i8nzVV/wg436pLE0K0wA0yjNqaxezP/BSspxjkkL7y99hD k= X-Google-Smtp-Source: AGHT+IETHWFelk6Mub34hbl75oMBkoJ65akYwAcLu4qq51tBseIZYkZ98qMqGR2LU1JqHBfdJjkCSmc1 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1682:b0:d7b:8d0c:43f1 with SMTP id bx2-20020a056902168200b00d7b8d0c43f1mr306276ybb.9.1694528331394; Tue, 12 Sep 2023 07:18:51 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:29 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3756; i=ardb@kernel.org; h=from:subject; bh=PA9RNauLEmhgmumF9kAUNop/3oYCQfl1LaCcmXQBZZk=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaP8hlSlMZ+Jn17Pv0N2ls3jduZRTonPezgqqmX2aS yS2riCro5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAExk6nmG32yZGYcerLre2fHq 3fMvBdqqLvmFxyYcLH6rpsO/4KXpgypGhjv/u32t2j4U7Ev//PhmhTE/t1dY+cQbm0N0FExik69 68QIA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-102-ardb@google.com> Subject: [PATCH v4 39/61] arm64: Revert "mm: provide idmap pointer to cpu_replace_ttbr1()" From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071853_016438_649AC1B9 X-CRM114-Status: GOOD ( 15.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel This reverts commit 1682c45b920643c, which is no longer needed now that we create the permanent kernel mapping directly during early boot. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/mmu_context.h | 13 ++++--------- arch/arm64/kernel/cpufeature.c | 2 +- arch/arm64/kernel/suspend.c | 2 +- arch/arm64/mm/kasan_init.c | 4 ++-- 4 files changed, 8 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index bea13dc4469f..e1eacba3f757 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -108,18 +108,13 @@ static inline void cpu_uninstall_idmap(void) cpu_switch_mm(mm->pgd, mm); } -static inline void __cpu_install_idmap(pgd_t *idmap) +static inline void cpu_install_idmap(void) { cpu_set_reserved_ttbr0(); local_flush_tlb_all(); cpu_set_idmap_tcr_t0sz(); - cpu_switch_mm(lm_alias(idmap), &init_mm); -} - -static inline void cpu_install_idmap(void) -{ - __cpu_install_idmap(idmap_pg_dir); + cpu_switch_mm(lm_alias(idmap_pg_dir), &init_mm); } /* @@ -150,7 +145,7 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, * avoiding the possibility of conflicting TLB entries being allocated. */ -static inline void cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap) +static inline void cpu_replace_ttbr1(pgd_t *pgdp) { typedef void (ttbr_replace_func)(phys_addr_t); extern ttbr_replace_func idmap_cpu_replace_ttbr1; @@ -174,7 +169,7 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap) replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); - __cpu_install_idmap(idmap); + cpu_install_idmap(); /* * We really don't want to take *any* exceptions while TTBR1 is diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index c91f658b175e..0014645c5417 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -3381,7 +3381,7 @@ subsys_initcall_sync(init_32bit_el0_mask); static void __maybe_unused cpu_enable_cnp(struct arm64_cpu_capabilities const *cap) { - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1(lm_alias(swapper_pg_dir)); } /* diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c index 0fbdf5fe64d8..7c2391851db6 100644 --- a/arch/arm64/kernel/suspend.c +++ b/arch/arm64/kernel/suspend.c @@ -55,7 +55,7 @@ void notrace __cpu_suspend_exit(void) /* Restore CnP bit in TTBR1_EL1 */ if (system_supports_cnp()) - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1(lm_alias(swapper_pg_dir)); /* * PSTATE was not saved over suspend/resume, re-enable any detected diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 856bda5fa19d..82798dcd6323 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -220,7 +220,7 @@ static void __init kasan_init_shadow(void) */ memcpy(tmp_pg_dir, swapper_pg_dir, sizeof(tmp_pg_dir)); dsb(ishst); - cpu_replace_ttbr1(lm_alias(tmp_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1(lm_alias(tmp_pg_dir)); clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); @@ -256,7 +256,7 @@ static void __init kasan_init_shadow(void) PAGE_KERNEL_RO)); memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE); - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1(lm_alias(swapper_pg_dir)); } static void __init kasan_init_depth(void) From patchwork Tue Sep 12 14:16:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 86F91CA0EF2 for ; Tue, 12 Sep 2023 15:30:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=nODrTtYiyJbWJhr5HU65zZ5pDg0QqzIemxURCtF/uN8=; b=cbR10GjmRmu00jOuV47VXhtvV+ p+HzALFzrplSQhGY+hmut5bzN6qG7FOoucBwDuULVJksoemUTmYHBkIoMIxYwPJCRgWiPVE+yznqH B3juyQzXQaFkb5hClNzcs3zy7jlPXFDqxIyXQYr4GQR6nFUjR/7XQPJli/lLkXUkDVhGy22ZawDmu LCuQOcH+oF+8EeF4M96G9RnvdOKxGwJX/j77sbLlieuJtVGb5IzSu9p7mqVuGBh8UPLVIl531hwa+ U3gzXGji5BkxEqnnwj6Y0uVRpMSSgryo5It6n7vkNJZ8b+CdoFABhFcUw2LO5as0R8s2rrO2c+14p c5eXc83A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg5LQ-003ih1-2U; Tue, 12 Sep 2023 15:30:28 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4EC-003X1r-0V for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:10 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-31fb896a87aso640895f8f.1 for ; Tue, 12 Sep 2023 07:18:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528334; x=1695133134; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=D6wn7wzrJQFrtWJKmHhlzm09wnf150CyBxnVvC6CXi0=; b=13/2x/HN9JXPTkMPqAU+8f11/H+hOxBZZyFZEkIqOpHq82A2eam0N6eFceWQeUKjz9 S09FWooKkKhz99jpljyzlhd6rokPU/7gGtVMQx6axz5MLmj+01xkqiScxd2rB5OvEHg0 znYiQDc9JwijcHRSXp6668PjUlo+e1e4tJ83dqSZXGclIhPOLwDZ/c53dFYl4mJySrfJ Gm6v2JihZmlo1XQvNsiDg0dDBp5WSB1GqknGT2JDxsL6ElDdzXHEhKAO1jt5zPZ6rofo vNg5N8MtutweEXwTZ6qlTPx59GpkdSpaStfWV9qRay3tUDyThOAYLvzG/VQJ2PbA0408 kajw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528334; x=1695133134; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=D6wn7wzrJQFrtWJKmHhlzm09wnf150CyBxnVvC6CXi0=; b=DDZwHS3uxnm33SmpDxf8u1ZbsXyJVRe8gr/HLcXAii6tplCCmjO0KUL3E191zY2bWy fR1AvlOtC1HdwRPHqbJlZxzeR3FMaDC3w77F74sCiWJn7YHlNDIs5l3fqlEw8bfoU1rf zMqUYLJzS/zdN7ihVaUyhEyHHRp2e2IxqTYewPHIQgvxQUFNZ9QE6nY6qKy9xlbl82rc DcmH9NZswos+3VMNed2NOd+64X4APJRbr/nB3nhDzesoBUZ2uLJYfNfu+eWHoc6P9ubj MYp6GVnnhOc1/zBgQoOljZRTFxv6EXOX6EAvYz4zC/8Ck3mGm7WsLfyBX4iYgqxWEzmZ v2Gg== X-Gm-Message-State: AOJu0YzYvs/8LuMkB3Wpe3qRSlCJcknBybTc7qfmAR7P8JH8re53l3wV RT7zdPAripblEJG+duz+K1c8MVn0BtA9DShMoUqjeI3lMi89Ii9THlwXC6nSjFg729Cimhafd0S B5Zy/KuI1B2TCozKuw10sOiN3IuMZOmVUdfDvC6VgLDV3NL8Mir097rrvoNT1VlaYzffDMzNWOp M= X-Google-Smtp-Source: AGHT+IG1jVwAOIllF7cHvBPNchNYw7Q6tQUWhzHdwAEYMplrl8anKApw08N01FXjCHutkrXSkPsXhKTd X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:ee8e:0:b0:31a:d2cf:c3d3 with SMTP id b14-20020adfee8e000000b0031ad2cfc3d3mr143114wro.14.1694528333546; Tue, 12 Sep 2023 07:18:53 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:30 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=4270; i=ardb@kernel.org; h=from:subject; bh=4a+6ZhkBs7eMY4F3SGokV+yqOw6iTijNai7SnmrXcuY=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6EDVpi9iLSEL5knbrls+79Fz1tUXFt178fLY1dNSE j0XL6Q/6ChlYRDjYJAVU2QRmP333c7TE6VqnWfJwsxhZQIZwsDFKQATcXjJ8N81Wczp6Psqxslm i/Ku5UVvNNis27k9qld6Z2iOyT7/3YIMfyWiVqonV0+tvSwocz3n2z6W+NPPlL8vFbit29gg++3 GRh4A X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-103-ardb@google.com> Subject: [PATCH v4 40/61] arm64: mmu: Make cpu_replace_ttbr1() out of line From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071856_222971_EE83DAA5 X-CRM114-Status: GOOD ( 19.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel cpu_replace_ttbr1() is a static inline, which means it gets instantiated wherever it is used. This is not really necessary, as it is never called on a hot path. It also has the unfortunate side effect that the symbol idmap_cpu_replace_ttbr1 may never be referenced from kCFI enabled C code, and this means the type id symbol may not exist either. This will result in a build error. (Note that this requires CnP, KAsan and suspend/resume to be disabled in the Kconfig but that is a valid config nonetheless). So let's just move it out of line so all callers will share the same implementation, which will reference idmap_cpu_replace_ttbr1 unconditionally. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/mmu_context.h | 41 +------------------- arch/arm64/mm/mmu.c | 41 ++++++++++++++++++++ 2 files changed, 42 insertions(+), 40 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index e1eacba3f757..4eeb460d7ed6 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -141,46 +141,7 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) isb(); } -/* - * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, - * avoiding the possibility of conflicting TLB entries being allocated. - */ -static inline void cpu_replace_ttbr1(pgd_t *pgdp) -{ - typedef void (ttbr_replace_func)(phys_addr_t); - extern ttbr_replace_func idmap_cpu_replace_ttbr1; - ttbr_replace_func *replace_phys; - unsigned long daif; - - /* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ - phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp)); - - if (system_supports_cnp() && !WARN_ON(pgdp != lm_alias(swapper_pg_dir))) { - /* - * cpu_replace_ttbr1() is used when there's a boot CPU - * up (i.e. cpufeature framework is not up yet) and - * latter only when we enable CNP via cpufeature's - * enable() callback. - * Also we rely on the system_cpucaps bit being set before - * calling the enable() function. - */ - ttbr1 |= TTBR_CNP_BIT; - } - - replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); - - cpu_install_idmap(); - - /* - * We really don't want to take *any* exceptions while TTBR1 is - * in the process of being replaced so mask everything. - */ - daif = local_daif_save(); - replace_phys(ttbr1); - local_daif_restore(daif); - - cpu_uninstall_idmap(); -} +void cpu_replace_ttbr1(pgd_t *pgdp); /* * It would be nice to return ASIDs back to the allocator, but unfortunately diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 1f74faffa408..e921eb643349 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1438,3 +1438,44 @@ void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte { set_pte_at(vma->vm_mm, addr, ptep, pte); } + +/* + * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, + * avoiding the possibility of conflicting TLB entries being allocated. + */ +void cpu_replace_ttbr1(pgd_t *pgdp) +{ + typedef void (ttbr_replace_func)(phys_addr_t); + extern ttbr_replace_func idmap_cpu_replace_ttbr1; + ttbr_replace_func *replace_phys; + unsigned long daif; + + /* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ + phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp)); + + if (system_supports_cnp() && !WARN_ON(pgdp != lm_alias(swapper_pg_dir))) { + /* + * cpu_replace_ttbr1() is used when there's a boot CPU + * up (i.e. cpufeature framework is not up yet) and + * latter only when we enable CNP via cpufeature's + * enable() callback. + * Also we rely on the system_cpucaps bit being set before + * calling the enable() function. + */ + ttbr1 |= TTBR_CNP_BIT; + } + + replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); + + cpu_install_idmap(); + + /* + * We really don't want to take *any* exceptions while TTBR1 is + * in the process of being replaced so mask everything. + */ + daif = local_daif_save(); + replace_phys(ttbr1); + local_daif_restore(daif); + + cpu_uninstall_idmap(); +} From patchwork Tue Sep 12 14:16:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9A15FCA0EEC for ; Tue, 12 Sep 2023 14:20:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=1ctFs0dKMiz93Do0OVXRDndkepkZBqTX8QbruCpQhc4=; b=dTBZ5NoL/CJAXKGrCgsEXqvIIh qYFTX0E/S5Iocs2NZMRwkJ+fQj1wUsZXCUlM2qwpCSdBWinWPIccD9uUJ4Wd5CNkIRKh3pufvFKy/ o0Dtp5myJcsZsqpt+6nxrhU20Rf7EWlo9E6AtuRQ8BwnEdaa0mdYgW8TkTqhYriOwiwHK1Z388GR/ TXczCdo0q27D6ybFTDNI2Gz/cFDteUuO8st7CQRgN7AYNvYv2xMmQTXEKwPDObdKds04LcOZDBNQf gQGy+20mOC2wruhZhls2NIFaXOKa2BRtIge3KyWzrJduhLAX3XzYamyjr61k/hd+WSB8Bsd6NxplJ XnZmAb3w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4FO-003Y4Y-0k; Tue, 12 Sep 2023 14:20:10 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4EE-003X3l-0y for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:11 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-31c470305cfso3775858f8f.3 for ; Tue, 12 Sep 2023 07:18:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528336; x=1695133136; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pk+vPXY/vt1uZZhvwQqemQR8yzBJaxiopgo6eUe4pps=; b=nU2gDS6KTvljmAZyhQfkaUTqMIGlyJIwO6z7Bd1i7K4gzLXLsjXhNC/rWEDg8YCKTr McWKPvpxlBOgRuTZq8X2RQ2JHCajKtXiBNrdFKlfnRMC734gSLsCJo8asXL5OD26Byz8 QgEeNkBnJfpTM4C2FWCryW70/d2rnlOOG8cjTcGkUG/BQmFkmxNcPLvSQyP/FgZPXij5 wtQA+/uD5HxyD/S3m3cTWvmQBAh4CAvQQOLB/bZfN09+TBWdq7j1Zzv2fpQmdr6s2wtP k22XOChI+AySY7eOLtJTFcEA1yFOzdbDSc8s28Q6ZJlmI1Wlo6+wCDeKdaWGjM3LNaQZ BY8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528336; x=1695133136; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pk+vPXY/vt1uZZhvwQqemQR8yzBJaxiopgo6eUe4pps=; b=W+rqZBpD2PM7+IEjhM2tOc2+l/qmPMUuZt2gOlB4wZna1K1XisgavlNBzfIL+y/8Hm 4ncz5EXja+gPpofb0lGNCd5fjTiiIY21JHQqmDmQ8zKQPznW6xRt7D+5SleeSfh1MqfZ U43zUd6cu43gXqUTh0vdGXyF7WCRJn+f4WsgvzxPdv+6b7ylosPmy6W8BIIXoypCw8np O9/IWe18DQRe4l2Neo/TPii3e6rt+CLt5PPNzkyif+enk+QTSd2LvENOJSfjI8JeaaRT EFIPOspvGX/hSd4J0MY+fMf6ttCVv+weePybAt+X7er8xxbrDDoE8KBjABtxsSOVLmQ2 4Mdw== X-Gm-Message-State: AOJu0YwqHIfaOSPpkJ7Wsrud8bbdXjxpAj6T+MpsJF+3Fz5O9qPbqgVh 6GpX4buhRCbROsUnHQ0W3twDfCrnti1lUELLqbhHSLNb9eMCdFFDyb4hDHjU/uiMkH8qj7nywVJ l3zshxCr/R3L2bfnQ3+I5Qh52aZTN/tSxFTUyrvEVViSXxHdOIwW8taBRf7Zlp1P4ebwAgtTJSP c= X-Google-Smtp-Source: AGHT+IGg6ykiQ4Itro8tKHaGRzxYx6FXwa2BBjEsWsP1Qa0mdwznPFPPsjVx+GBaT/5uf/x8XSjpkjiC X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:ce86:0:b0:31c:6629:a583 with SMTP id r6-20020adfce86000000b0031c6629a583mr142030wrn.13.1694528335745; Tue, 12 Sep 2023 07:18:55 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:31 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=974; i=ardb@kernel.org; h=from:subject; bh=3RIbQ+ir09pn35apY5Qp22tjlJ+wR04ahdGMYRgLFlQ=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6CBz6NI72rFW96NPHknxcpii8US6ZWK672/3PqMVF 3i21Gt1lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgIn08TH8rzBU2F1pIHO7tOmb nSpnRFD/I9ZVq2b5GMkd1pIMXXT4OMNfyXRxy01Cj1wDZ+549dzCaO2rZF9e/9YPHfvF9n6rb5j AAgA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-104-ardb@google.com> Subject: [PATCH v4 41/61] arm64/mm: Add FEAT_LPA2 specific TCR_EL1.DS field From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071858_422964_0C820B60 X-CRM114-Status: GOOD ( 11.23 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Anshuman Khandual As per ARM ARM (0487G.A) TCR_EL1.DS fields controls whether 52 bit input and output address get supported on 4K and 16K page size configuration, when FEAT_LPA2 is known to have been implemented. This adds TCR_DS field definition which would be used when FEAT_LPA2 gets enabled. Acked-by: Catalin Marinas Signed-off-by: Anshuman Khandual Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/pgtable-hwdef.h | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index e4944d517c99..b770f98fc0b5 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -284,6 +284,7 @@ #define TCR_E0PD1 (UL(1) << 56) #define TCR_TCMA0 (UL(1) << 57) #define TCR_TCMA1 (UL(1) << 58) +#define TCR_DS (UL(1) << 59) /* * TTBR. From patchwork Tue Sep 12 14:16:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5675ACA0EEC for ; Tue, 12 Sep 2023 14:20:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Y0rb2VeZIDVJBxDHfiQL0fsoAgfYFT7iRwzGRpgsiz4=; b=DFbP3enQse2I/n5kGcgcTL5iO0 /dJGUGBN48x/zv7FtrC1Sl0misIZbnwQpenNNBgJSq7JVLd1zHvzZL2ouZ2VYkmr3rumUARn1y6wl 5PncOFm2OVKzAgDefZQ659vp+LC/sdgJ0/cVdip+cZOGb1GWYeZEN5m4m6Kzxbi9e7Uv6GRrnHgmY FLAVa39J2BL1b4Cs6Vs5eD/QYCJPk8bVp4otseOhpikBPY3hVlRJGf0pRKgwNbmaLhu2BB8IQAZZv lh5294DldoYBFdk1wigpSTZMxH3P6MxMMTZsxBjle0se+MXn91RjWGPK07UCHj4rPEIuawBuuuG5H g/CQSqQw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4FU-003Y9d-0O; Tue, 12 Sep 2023 14:20:16 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4EF-003X60-33 for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:11 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-58cb845f2f2so62109827b3.1 for ; Tue, 12 Sep 2023 07:18:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528338; x=1695133138; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NmxYtpZy7xid9+3ZXEZ7J6YHnsxvGYTkj2THnUNOVfw=; b=2IL6Jcs/F7qouhQemy/WPmPvKvejeSgMSm5uAXtqXSUMtOQQTPYp+3v1m3f2waDg0N IiGJZVHN5UIL0LTgEv63SS/JMsnV6oHe7M0x5gnWGNDz4eYmffdE5TlNB5BMNuvkZd3f pLFCR+3T71wTEW01jeXhBpC4ucT1UzF0Pg+CbF5MZkVEogb1zhJA4mT6OE9EY7QlPiyG TqMc0vjBx3jKj25ag2vsALaEod50eZM1tmre+t2Lhsx17BmlVVIqLtG7SHz+kxTLUTpW kp3lwCRMwA1Hc17FP8ISMPMJsCYqJWH73SPrIV0pe/t8WVYmZQlU/0shNtJJX0iy+IgW yATg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528338; x=1695133138; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NmxYtpZy7xid9+3ZXEZ7J6YHnsxvGYTkj2THnUNOVfw=; b=FZpaDTvGz7BsdF52zrEycVQJeo3A14ASMCZcdm9wp/8ooDogiY5qbzvu7dXE61sRtg 9R/gYO/tXjeGkyHSPWFkgpTA+8FISla8sRsor8dsLBJNt7VNm6YAczXDEheuPW+LvsZP 9ky1fTwSMz+AMCjBgsuBDWrTHbiHFzp7JauAaJE44G7x7OebuiTIQ24hdCLLlGjyhE3Y GS9bxFAs8bUc9bzr4K5MvMnKwh1jbsvyAbxvGEhjD/FOLBJWNYr9Tj2x75uIpwpBHMv9 7g5WNRzBgBsvAkX1+kdtmI65xE66Rr2zRttAYn4ZdTfsC8AS0rwaWkFIa4c1MmgY6I1p GyuA== X-Gm-Message-State: AOJu0Yzulu/YFp57RQ9ygtCreV+/6inGoSQLZGIexD+p88glXzAB1loz 4NeCK86eV5X97SI/MfpwUjv85joakeuo+Y4PhPs0kGpe3yNRBPHfV8H84hzWbnqjlaM4y4auIEB 50eWVziBNb3fS+vcwGr00+IyppXTz0Z2E5QmVjFrUqg6unqxkAIk1Tqp0LMEuKYeOW0t50iWmOn 4= X-Google-Smtp-Source: AGHT+IG/hCVt/iMkYf9UOLEx6RANZb0fn3OzFjD9H/1MyJhDJJgZaoS7l7Y97AIDBnrjhEN+ZUZ43W4q X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:ed01:0:b0:589:a935:b13 with SMTP id k1-20020a81ed01000000b00589a9350b13mr322197ywm.5.1694528338152; Tue, 12 Sep 2023 07:18:58 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:32 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1741; i=ardb@kernel.org; h=from:subject; bh=1zwjYCdptLGI5sQYr1Z4N+eAr5O5DnPs7MuLg236ZgE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6JBgiUm+n9NaG50PORO/tvP4nml7sKW2Jm5bhS7rR P+4h0c7SlkYxDgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwER+rGL4H/t+R+iG6arr9p4p Cd388M3mdTuXl8zMDuC9s+Joi2x/cj/Df6fZ0/Zofyub4384c0q5r/S+svO5Vs+OnDlheHZpxuS NjxgA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-105-ardb@google.com> Subject: [PATCH v4 42/61] arm64/mm: Add FEAT_LPA2 specific ID_AA64MMFR0.TGRAN[2] From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071900_014565_BD706CED X-CRM114-Status: GOOD ( 10.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Anshuman Khandual PAGE_SIZE support is tested against possible minimum and maximum values for its respective ID_AA64MMFR0.TGRAN field, depending on whether it is signed or unsigned. But then FEAT_LPA2 implementation needs to be validated for 4K and 16K page sizes via feature specific ID_AA64MMFR0.TGRAN values. Hence it adds FEAT_LPA2 specific ID_AA64MMFR0.TGRAN[2] values per ARM ARM (0487G.A). Acked-by: Catalin Marinas Signed-off-by: Anshuman Khandual Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/sysreg.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 38296579a4fd..a567add35301 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -847,11 +847,13 @@ #if defined(CONFIG_ARM64_4K_PAGES) #define ID_AA64MMFR0_EL1_TGRAN_SHIFT ID_AA64MMFR0_EL1_TGRAN4_SHIFT +#define ID_AA64MMFR0_EL1_TGRAN_LPA2 ID_AA64MMFR0_EL1_TGRAN4_52_BIT #define ID_AA64MMFR0_EL1_TGRAN_SUPPORTED_MIN ID_AA64MMFR0_EL1_TGRAN4_SUPPORTED_MIN #define ID_AA64MMFR0_EL1_TGRAN_SUPPORTED_MAX ID_AA64MMFR0_EL1_TGRAN4_SUPPORTED_MAX #define ID_AA64MMFR0_EL1_TGRAN_2_SHIFT ID_AA64MMFR0_EL1_TGRAN4_2_SHIFT #elif defined(CONFIG_ARM64_16K_PAGES) #define ID_AA64MMFR0_EL1_TGRAN_SHIFT ID_AA64MMFR0_EL1_TGRAN16_SHIFT +#define ID_AA64MMFR0_EL1_TGRAN_LPA2 ID_AA64MMFR0_EL1_TGRAN16_52_BIT #define ID_AA64MMFR0_EL1_TGRAN_SUPPORTED_MIN ID_AA64MMFR0_EL1_TGRAN16_SUPPORTED_MIN #define ID_AA64MMFR0_EL1_TGRAN_SUPPORTED_MAX ID_AA64MMFR0_EL1_TGRAN16_SUPPORTED_MAX #define ID_AA64MMFR0_EL1_TGRAN_2_SHIFT ID_AA64MMFR0_EL1_TGRAN16_2_SHIFT From patchwork Tue Sep 12 14:16:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381800 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 99CBFCA0EEB for ; Tue, 12 Sep 2023 14:20:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=rdHfVX0xkvg286/2UAYe22BbOony+H3o0koaq2eWgKI=; b=xwTGsleLFczaKvUIBLd0oBnqHo eL6CqfsxeljVWCVkmhYC4k7btJAvJ8l5Armzw+friPdeaG1u9EQhdgafpVwCT2a2oFfXRW6fRmVF0 qfC0I53jaM5U6CRFqvWg8GCD0lPU0juU5Atzkt59c0fnzY5xfa2XJHvekBM5A57m+9GiJlyp1F+8t RA3rjYyxLypU/HFPf0qLjP7ImnR1KeYJvk5XH5PxreJfgs7yV8Pt+y5939cToJ7SboORCRQ4qB6EW 6Rm94bDW5aGQZTWSvH5m04Q64bjTDHLioIYTslp5Ze6HSqS83qKPJvhZ5BO7pxPFui30CbbSEWSO6 kpV6DtJA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4FB-003XtW-0F; Tue, 12 Sep 2023 14:19:57 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4EO-003XDX-2v for linux-arm-kernel@bombadil.infradead.org; Tue, 12 Sep 2023 14:19:09 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=85LYl4rNmCmAOntagah2l0TFvaw+wEpPhP3xYyATt24=; b=lBn743i4AmKSXuAEcWRINNf0vM G3WAuh9DGqcuubXnIChKG6Yb8KB5AMflQpB4Bjl3L1geJzjiX5MX0/XeJLkQMb1lJ7Rq7kDt2NV1E srMm0ykgZ3Vdgf0XxDRfUQBwdrGp1pawOd/9gVNZfOIK281qaaNoKJHR2JmIEzJUA2ZEYLnou/amG VDJV4TCvGy5dEPBIRnsrINL7spUS3RI8wYbnxv6Grazj/Clmn8lkkifnx7BAbZu6aZXiqYLEoPBqU gZQIPjas+rIzsB5YNmJbLl90V/oJBz6K2vg4fZuoRbREF1Ana9xeitzdu22S+TOheYUX+lbALStoU q0vk7WgA==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4EH-006A1z-0j for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:07 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-58c8b2d6784so63639717b3.3 for ; Tue, 12 Sep 2023 07:19:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528340; x=1695133140; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=85LYl4rNmCmAOntagah2l0TFvaw+wEpPhP3xYyATt24=; b=oWo5bR1EYQg5Cd9yHjM9+TXmBNslT08qMLRDdjiNu1ubDOvvh4ESHHGgid1CdmCHTu pTC2UuBppA+GKljsUxea+QMtUomenp8hWH4hYOH17o/OnxrdRK9VjbcvG8T1PBW83Wf9 cRUOGp4uvuMIPBJlKgC0qnE43tHvAKBDgbgsbtAB7cCLEbALeVe5yFYzcZZ+rFfVFYDq Oo7SmWl7UzEaUX7fKkz29HDXMGwFd2h/18twW2mKZKfhrSOb+CnF1fgDi/yUuBnzRkm1 N6xIrKSUFvFgBiiwtCY79AiE5kpTI+VtWv0/8OQIbre/e/Adrnot+4GFXraeZHqhNnXe 4RDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528340; x=1695133140; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=85LYl4rNmCmAOntagah2l0TFvaw+wEpPhP3xYyATt24=; b=iK/zo1YYb6jIb10gE3FCT1XoOLEzmHIsR5MwTFlQCkVghe1K8Jp3t2++nVtDLAmd+1 uQGyioMcv3Wt/mwB+jFRAsSGUUqI3C32k/5YYwIrsvzZelTdpA8R7QboXw95Ik9nRCVC fyGcQxjlpt/0lU1fOhx9cBS0VdOXDAWFthGJvXANZ2rmYxcvAZ7Y6coJAJorWHGNkBTT +MQzoYlllyK/T1RqZgDpCKJvbV6bn3rsrPOVpUByMG98DuVX97OYLFUL5etBtNEm/Kwx Xg5/cqFTTT2f/zRZcM9fgkW/HvQggJBxddeK80x/0JCjF5WcZDGKznLjBV9n71c+gzIv SkwA== X-Gm-Message-State: AOJu0YxTtmIxvlStO1IMl5V7cMHnP+GlkFvD3mhFN+S+cOnrj8b8LR32 k2s4fxEwClC/KBxyn0aqaEi16sYPZPjg3BxJpqEHvm3Mou37X2zryGmjz1r6IPUAbyMtsllSEC7 lNqyN7DPIrts8v1WNpmvOu3LSdChRw1qJYP0o0Sg32vcjLW/EKVcN5jdI9/GIAqiaUiHBhoqfU1 8= X-Google-Smtp-Source: AGHT+IEl0qRwaL28zgR5Rzo2qjbuKRexPp65Ac9GvKB/3M+6CaWdmfJvS9qx/mOKMesVpVktyLBrjYLQ X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:ad48:0:b0:595:39a:5473 with SMTP id l8-20020a81ad48000000b00595039a5473mr349885ywk.10.1694528340450; Tue, 12 Sep 2023 07:19:00 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:33 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10209; i=ardb@kernel.org; h=from:subject; bh=PPvSQjUa/V924GxEdaI5ksTzPK2C8LGSIUfpV57ERv8=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6PC/Tz/8VxydcvSK6x6lmK6w1m8SDLJFRstrslI05 5z6vG1PRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZhIiiQjwyENnR8TDBdw5X5+ 7cY568Kaq3qHdvC+qNNge2dQs6eT4Q/DHy4O0z0yT47f3y7kpORqW/pnZlv7FMOHp6/9r8rYKBS dxwAA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-106-ardb@google.com> Subject: [PATCH v4 43/61] arm64: mm: Handle LVA support as a CPU feature From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_151902_804427_E4BD0BA8 X-CRM114-Status: GOOD ( 25.93 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Currently, we detect CPU support for 52-bit virtual addressing (LVA) extremely early, before creating the kernel page tables or enabling the MMU. We cannot override the feature this early, and so large virtual addressing is always enabled on CPUs that implement support for it if the software support for it was enabled at build time. It also means we rely on non-trivial code in asm to deal with this feature. Given that both the ID map and the TTBR1 mapping of the kernel image are guaranteed to be 48-bit addressable, it is not actually necessary to enable support this early, and instead, we can model it as a CPU feature. That way, we can rely on code patching to get the correct TCR.T1SZ values programmed on secondary boot and resume from suspend. On the primary boot path, we simply enable the MMU with 48-bit virtual addressing initially, and update TCR.T1SZ if LVA is supported from C code, right before creating the kernel mapping. Given that TTBR1 still points to reserved_pg_dir at this point, updating TCR.T1SZ should be safe without the need for explicit TLB maintenance. Since this gets rid of all accesses to the vabits_actual variable from asm code that occurred before TCR.T1SZ had been programmed, we no longer have a need for this variable, and we can replace it with a C expression that produces the correct value directly, based on the value of TCR.T1SZ. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 9 ++++++ arch/arm64/include/asm/memory.h | 13 ++++++++- arch/arm64/kernel/cpufeature.c | 13 +++++++++ arch/arm64/kernel/head.S | 29 +++++--------------- arch/arm64/kernel/image-vars.h | 1 - arch/arm64/kernel/pi/map_kernel.c | 3 ++ arch/arm64/kernel/sleep.S | 3 -- arch/arm64/mm/mmu.c | 5 ---- arch/arm64/mm/proc.S | 9 +++--- arch/arm64/tools/cpucaps | 1 + 10 files changed, 49 insertions(+), 37 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 128861e5d32d..4ffbd08f6b77 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -977,6 +977,15 @@ static inline bool cpu_has_pac(void) return feat; } +static inline bool cpu_has_lva(void) +{ + u64 mmfr2; + + mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + return cpuid_feature_extract_unsigned_field(mmfr2, + ID_AA64MMFR2_EL1_VARange_SHIFT); +} + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index b49575a92afc..f6277f66388f 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -183,9 +183,20 @@ #include #include #include +#include + +static inline u64 __pure read_tcr(void) +{ + u64 tcr; + + // read_sysreg() uses asm volatile, so avoid it here + asm("mrs %0, tcr_el1" : "=r"(tcr)); + return tcr; +} #if VA_BITS > 48 -extern u64 vabits_actual; +// For reasons of #include hell, we can't use TCR_T1SZ_OFFSET/TCR_T1SZ_MASK here +#define vabits_actual (64 - ((read_tcr() >> 16) & 63)) #else #define vabits_actual ((u64)VA_BITS) #endif diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 0014645c5417..3778a7384c62 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2678,6 +2678,19 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_cpuid_feature, ARM64_CPUID_FIELDS(ID_AA64MMFR2_EL1, EVT, IMP) }, +#ifdef CONFIG_ARM64_VA_BITS_52 + { + .desc = "52-bit Virtual Addressing (LVA)", + .capability = ARM64_HAS_VA52, + .type = ARM64_CPUCAP_BOOT_CPU_FEATURE, + .sys_reg = SYS_ID_AA64MMFR2_EL1, + .sign = FTR_UNSIGNED, + .field_width = 4, + .field_pos = ID_AA64MMFR2_EL1_VARange_SHIFT, + .matches = has_cpuid_feature, + .min_field_value = ID_AA64MMFR2_EL1_VARange_52, + }, +#endif {}, }; diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 545b5d8976f4..e25351addfd0 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -80,7 +80,6 @@ * x19 primary_entry() .. start_kernel() whether we entered with the MMU on * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 - * x25 primary_entry() .. start_kernel() supported VA size */ SYM_CODE_START(primary_entry) bl record_mmu_state @@ -125,14 +124,6 @@ SYM_CODE_START(primary_entry) * On return, the CPU will be ready for the MMU to be turned on and * the TCR will have been set. */ -#if VA_BITS > 48 - mrs_s x0, SYS_ID_AA64MMFR2_EL1 - tst x0, ID_AA64MMFR2_EL1_VARange_MASK - mov x0, #VA_BITS - mov x25, #VA_BITS_MIN - csel x25, x25, x0, eq - mov x0, x25 -#endif bl __cpu_setup // initialise processor b __primary_switch SYM_CODE_END(primary_entry) @@ -242,11 +233,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) mov x0, x20 bl set_cpu_boot_mode_flag -#if VA_BITS > 48 - adr_l x8, vabits_actual // Set this early so KASAN early init - str x25, [x8] // ... observes the correct value - dc civac, x8 // Make visible to booting secondaries -#endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif @@ -376,10 +362,13 @@ SYM_FUNC_START_LOCAL(secondary_startup) * Common entry point for secondary CPUs. */ mov x20, x0 // preserve boot mode + +#ifdef CONFIG_ARM64_VA_BITS_52 +alternative_if ARM64_HAS_VA52 bl __cpu_secondary_check52bitva -#if VA_BITS > 48 - ldr_l x0, vabits_actual +alternative_else_nop_endif #endif + bl __cpu_setup // initialise processor adrp x1, swapper_pg_dir adrp x2, idmap_pg_dir @@ -482,12 +471,8 @@ SYM_FUNC_START(__enable_mmu) ret SYM_FUNC_END(__enable_mmu) +#ifdef CONFIG_ARM64_VA_BITS_52 SYM_FUNC_START(__cpu_secondary_check52bitva) -#if VA_BITS > 48 - ldr_l x0, vabits_actual - cmp x0, #52 - b.ne 2f - mrs_s x0, SYS_ID_AA64MMFR2_EL1 and x0, x0, ID_AA64MMFR2_EL1_VARange_MASK cbnz x0, 2f @@ -498,9 +483,9 @@ SYM_FUNC_START(__cpu_secondary_check52bitva) wfi b 1b -#endif 2: ret SYM_FUNC_END(__cpu_secondary_check52bitva) +#endif SYM_FUNC_START_LOCAL(__no_granule_support) /* Indicate that this CPU can't boot and is stuck in the kernel */ diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 2ae33cc02051..d312f4176a2c 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -34,7 +34,6 @@ PROVIDE(__pi___memcpy = __pi_memcpy); PROVIDE(__pi___memmove = __pi_memmove); PROVIDE(__pi___memset = __pi_memset); -PROVIDE(__pi_vabits_actual = vabits_actual); PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); PROVIDE(__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index d0fad93a6872..18a8d4249136 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -166,6 +166,9 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) chosen = fdt_path_offset(fdt, chosen_str); init_feature_override(boot_status, fdt, chosen); + if (VA_BITS > VA_BITS_MIN && cpu_has_lva()) + sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(VA_BITS)); + /* * The virtual KASLR displacement modulo 2MiB is decided by the * physical placement of the image, as otherwise, we might not be able diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S index 2aa5129d8253..f093cdf71be1 100644 --- a/arch/arm64/kernel/sleep.S +++ b/arch/arm64/kernel/sleep.S @@ -102,9 +102,6 @@ SYM_CODE_START(cpu_resume) mov x0, xzr bl init_kernel_el mov x19, x0 // preserve boot mode -#if VA_BITS > 48 - ldr_l x0, vabits_actual -#endif bl __cpu_setup /* enable the MMU early - so we can access sleep_save_stash by va */ adrp x1, swapper_pg_dir diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index e921eb643349..50972763a67e 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -45,11 +45,6 @@ #define NO_CONT_MAPPINGS BIT(1) #define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ -#if VA_BITS > 48 -u64 vabits_actual __ro_after_init = VA_BITS_MIN; -EXPORT_SYMBOL(vabits_actual); -#endif - u64 kimage_voffset __ro_after_init; EXPORT_SYMBOL(kimage_voffset); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 6d2de0996c00..2995a92940e0 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -397,8 +397,6 @@ SYM_FUNC_END(idmap_kpti_install_ng_mappings) * * Initialise the processor for turning the MMU on. * - * Input: - * x0 - actual number of VA bits (ignored unless VA_BITS > 48) * Output: * Return in x0 the value of the SCTLR_EL1 register. */ @@ -423,16 +421,17 @@ SYM_FUNC_START(__cpu_setup) mair .req x17 tcr .req x16 mov_q mair, MAIR_EL1_SET - mov_q tcr, TCR_T0SZ(IDMAP_VA_BITS) | TCR_T1SZ(VA_BITS) | TCR_CACHE_FLAGS | \ + mov_q tcr, TCR_T0SZ(IDMAP_VA_BITS) | TCR_T1SZ(VA_BITS_MIN) | TCR_CACHE_FLAGS | \ TCR_SMP_FLAGS | TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS tcr_clear_errata_bits tcr, x9, x5 #ifdef CONFIG_ARM64_VA_BITS_52 - sub x9, xzr, x0 - add x9, x9, #64 + mov x9, #64 - VA_BITS +alternative_if ARM64_HAS_VA52 tcr_set_t1sz tcr, x9 +alternative_else_nop_endif #endif /* diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index c3f06fdef609..8ad0e876f8a0 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -50,6 +50,7 @@ HAS_STAGE2_FWB HAS_TCR2 HAS_TIDCP1 HAS_TLB_RANGE +HAS_VA52 HAS_VIRT_HOST_EXTN HAS_WFXT HW_DBM From patchwork Tue Sep 12 14:16:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 89BB4CA0EEC for ; Tue, 12 Sep 2023 14:20:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=rB4qKVxW0HUr7z4abZoNLUtg/Jzt0iYuZQEdccXahEc=; b=fo/VRRkLdHOyR7Bl0XXFDOJtdq db/MpOTRjqRJPRa7NhN64ar4WTE5APC/Z40Us0S1NLIbC0zXX5G1qzsS1byjdlko+lhmzPPRqxIrf pZ4OBHwvIR5G5ITWldVql1lGjdeXFFmHkg4siGSFmvREtwMAMOy/wk+yEulUljmp/hSjWUa0ygvj0 2HVQuwHxGHz3e786/mth2XPAQk+jCJOUyi5QaV6XD5XZX7hld80i2fP+G44jiKGWBD7AXCfh06N7G 5n4opkwDokvCzT1O4AFnbJ3Dj1E+yLmaR9b2MTRjPKlnSbnUucwJjV5IxAAMgRC1IMRHAKZvWbRgF rIj1XEBg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4FX-003YBm-37; Tue, 12 Sep 2023 14:20:20 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4EK-003X9j-2z for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:15 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-d802ecb5883so3748470276.3 for ; Tue, 12 Sep 2023 07:19:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528343; x=1695133143; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7g3rQc5ohntznFdEpK61zpmbym33bi0tjP5Qy9B6T6w=; b=BTyhEZLPABJy3ov636XWkCV2z+1cLRNfaGcrcaoJYnuArIDdHLCdzk9yPqDeXCpFJ/ k2xfNkuK/W1ZawJHPlkphfX8aN3wYNaCMAHb6YMKILhELAOl3zo312VfEHe65DjXg0LA BLNYgm/2nKKhbHgj0sEPRIpmYNxTZrErvCExxQHzb6n5BZdXrF0VpFoQqiirT2jvff9v 2AfC9ww4hLT+iAzE3xuHo5gBO5htLCfYAEyl6faCKOh6EfZrCOqNS2WBX+Y6LYeyJWQT fx9BBjm1QBx2i5HrsRA9KsdAACjTXyzaOkxsg1tr1CfzBdLqEyYykMzeQbDIvSYxVFol ZO3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528343; x=1695133143; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7g3rQc5ohntznFdEpK61zpmbym33bi0tjP5Qy9B6T6w=; b=njkYYw4XDDZaf4BR/ze9pOkECPXJ97AI2jQWchRmwEVc2oIsm+60B/TqT3uBMliHX1 BpMyXQPqGm9M17Li7FJAdKS0YOpd656afHo4JTyMU2PLj8JnszbUXkFKsakMqdjN1fvP crnzWe5iO78OjWpQT8ytXXB991bPPhM3/3KjnbKKwkoh+bo6gEami6wK+EBuqjRv0YLP uLQXNfEMrAmHC5czhnLcvTvIiEU5jq3ibyd4hRGIWy8c7oaJ6ddepfz3u1Ol4151Y6Q2 kER5uSAdrkm0T+vklJois1vbVwTnj6W2um3ofqvgcwzX+R28Mg+DcGYLxRvHxVVOs7Gn FPMw== X-Gm-Message-State: AOJu0Yxeg5u4gxgUIpMUXQS546EC0eVZFLyeSSi/HHjKgPEVh84NEab3 SFWXTXjCp8gp3fqT50r3vI4BMe8GlfS8WTLkmSAYD0NCPHhOkpHG/Z2feO1o4G1yYoAm/jyXd1B +4x8f4C0hduOpnWhXidSLxCMEvfKv+vvaN9SwvWivCen4igTSvV9GIaRcDRNDksZuR886VvPH3z Y= X-Google-Smtp-Source: AGHT+IFfwkCsrpwwd99WKR7AW03+TpYam0LEMMgkUE98Z2CPsVrxOKylh4DfoGi+4q+jP/9x+i4GAm5g X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:adc3:0:b0:d77:f7c3:37db with SMTP id d3-20020a25adc3000000b00d77f7c337dbmr302647ybe.8.1694528342614; Tue, 12 Sep 2023 07:19:02 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:34 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6960; i=ardb@kernel.org; h=from:subject; bh=2dJHCIiitpBTpEMWV5ZMZiGNeeCX6/OF7FQfNtGJZlY=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6MiNa0es+BmvHtFul/fM8ZzNVDlt7SXdl8Yvply1W FC50JS9o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzk33SGP9wN4QxFrTYvKqJn WjveeTRlp2QGs8OutMNc66amvPu3YjYjw4lzYiKnhU2WFD+NfmLY8F2hdHc4Z/oby2fL3FIXXnL zYAEA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-107-ardb@google.com> Subject: [PATCH v4 44/61] arm64: mm: Add feature override support for LVA From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071905_000869_7334F874 X-CRM114-Status: GOOD ( 23.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add support for overriding the VARange field of the MMFR2 CPU ID register. This permits the associated LVA feature to be overridden early enough for the boot code that creates the kernel mapping to take it into account. Given that LPA2 implies LVA, disabling the latter should disable the former as well. So override the ID_AA64MMFR0.TGran field of the current page size as well if it advertises support for 52-bit addressing. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 17 ++++++----- arch/arm64/include/asm/cpufeature.h | 4 +++ arch/arm64/kernel/cpufeature.c | 8 +++-- arch/arm64/kernel/image-vars.h | 2 ++ arch/arm64/kernel/pi/idreg-override.c | 31 ++++++++++++++++++++ 5 files changed, 53 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index beb53bbd8c19..0710c17800a4 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -576,18 +576,21 @@ alternative_endif .endm /* - * Offset ttbr1 to allow for 48-bit kernel VAs set with 52-bit PTRS_PER_PGD. + * If the kernel is built for 52-bit virtual addressing but the hardware only + * supports 48 bits, we cannot program the pgdir address into TTBR1 directly, + * but we have to add an offset so that the TTBR1 address corresponds with the + * pgdir entry that covers the lowest 48-bit addressable VA. + * * orr is used as it can cover the immediate value (and is idempotent). - * In future this may be nop'ed out when dealing with 52-bit kernel VAs. * ttbr: Value of ttbr to set, modified. */ .macro offset_ttbr1, ttbr, tmp #ifdef CONFIG_ARM64_VA_BITS_52 - mrs_s \tmp, SYS_ID_AA64MMFR2_EL1 - and \tmp, \tmp, #(0xf << ID_AA64MMFR2_EL1_VARange_SHIFT) - cbnz \tmp, .Lskipoffs_\@ - orr \ttbr, \ttbr, #TTBR1_BADDR_4852_OFFSET -.Lskipoffs_\@ : + mrs \tmp, tcr_el1 + and \tmp, \tmp, #TCR_T1SZ_MASK + cmp \tmp, #TCR_T1SZ(VA_BITS_MIN) + orr \tmp, \ttbr, #TTBR1_BADDR_4852_OFFSET + csel \ttbr, \tmp, \ttbr, eq #endif .endm diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 4ffbd08f6b77..351e0b39f0bb 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -912,7 +912,9 @@ static inline unsigned int get_vmid_bits(u64 mmfr1) s64 arm64_ftr_safe_value(const struct arm64_ftr_bits *ftrp, s64 new, s64 cur); struct arm64_ftr_reg *get_arm64_ftr_reg(u32 sys_id); +extern struct arm64_ftr_override id_aa64mmfr0_override; extern struct arm64_ftr_override id_aa64mmfr1_override; +extern struct arm64_ftr_override id_aa64mmfr2_override; extern struct arm64_ftr_override id_aa64pfr0_override; extern struct arm64_ftr_override id_aa64pfr1_override; extern struct arm64_ftr_override id_aa64zfr0_override; @@ -982,6 +984,8 @@ static inline bool cpu_has_lva(void) u64 mmfr2; mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + mmfr2 &= ~id_aa64mmfr2_override.mask; + mmfr2 |= id_aa64mmfr2_override.val; return cpuid_feature_extract_unsigned_field(mmfr2, ID_AA64MMFR2_EL1_VARange_SHIFT); } diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 3778a7384c62..a2c320ed36b3 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -664,7 +664,9 @@ static const struct arm64_ftr_bits ftr_raz[] = { #define ARM64_FTR_REG(id, table) \ __ARM64_FTR_REG_OVERRIDE(#id, id, table, &no_override) +struct arm64_ftr_override id_aa64mmfr0_override; struct arm64_ftr_override id_aa64mmfr1_override; +struct arm64_ftr_override id_aa64mmfr2_override; struct arm64_ftr_override id_aa64pfr0_override; struct arm64_ftr_override id_aa64pfr1_override; struct arm64_ftr_override id_aa64zfr0_override; @@ -728,10 +730,12 @@ static const struct __ftr_reg_entry { &id_aa64isar2_override), /* Op1 = 0, CRn = 0, CRm = 7 */ - ARM64_FTR_REG(SYS_ID_AA64MMFR0_EL1, ftr_id_aa64mmfr0), + ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR0_EL1, ftr_id_aa64mmfr0, + &id_aa64mmfr0_override), ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR1_EL1, ftr_id_aa64mmfr1, &id_aa64mmfr1_override), - ARM64_FTR_REG(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2), + ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2, + &id_aa64mmfr2_override), ARM64_FTR_REG(SYS_ID_AA64MMFR3_EL1, ftr_id_aa64mmfr3), /* Op1 = 0, CRn = 1, CRm = 2 */ diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index d312f4176a2c..c5eeb33a6280 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -36,7 +36,9 @@ PROVIDE(__pi___memset = __pi_memset); PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); +PROVIDE(__pi_id_aa64mmfr0_override = id_aa64mmfr0_override); PROVIDE(__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); +PROVIDE(__pi_id_aa64mmfr2_override = id_aa64mmfr2_override); PROVIDE(__pi_id_aa64pfr0_override = id_aa64pfr0_override); PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override); PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index 5857e4efad59..062aeae68936 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -63,6 +63,35 @@ static const struct ftr_set_desc mmfr1 __prel64_initconst = { }, }; + +static bool __init mmfr2_varange_filter(u64 val) +{ + int __maybe_unused feat; + + if (val) + return false; + +#ifdef CONFIG_ARM64_LPA2 + feat = cpuid_feature_extract_signed_field(read_sysreg(id_aa64mmfr0_el1), + ID_AA64MMFR0_EL1_TGRAN_SHIFT); + if (feat >= ID_AA64MMFR0_EL1_TGRAN_LPA2) { + id_aa64mmfr0_override.val |= + (ID_AA64MMFR0_EL1_TGRAN_LPA2 - 1) << ID_AA64MMFR0_EL1_TGRAN_SHIFT; + id_aa64mmfr0_override.mask |= 0xfU << ID_AA64MMFR0_EL1_TGRAN_SHIFT; + } +#endif + return true; +} + +static const struct ftr_set_desc mmfr2 __prel64_initconst = { + .name = "id_aa64mmfr2", + .override = &id_aa64mmfr2_override, + .fields = { + FIELD("varange", ID_AA64MMFR2_EL1_VARange_SHIFT, mmfr2_varange_filter), + {} + }, +}; + static bool __init pfr0_sve_filter(u64 val) { /* @@ -173,6 +202,7 @@ static const union { prel64_t reg_prel; } regs[] __prel64_initconst = { { .reg = &mmfr1 }, + { .reg = &mmfr2 }, { .reg = &pfr0 }, { .reg = &pfr1 }, { .reg = &isar1 }, @@ -198,6 +228,7 @@ static const struct { { "arm64.nomte", "id_aa64pfr1.mte=0" }, { "nokaslr", "arm64_sw.nokaslr=1" }, { "rodata=off", "arm64_sw.rodataoff=1" }, + { "arm64.nolva", "id_aa64mmfr2.varange=0" }, }; static int __init parse_hexdigit(const char *p, u64 *v) From patchwork Tue Sep 12 14:16:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381806 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 22C01CA0EEC for ; Tue, 12 Sep 2023 14:20:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=krFINH6Ql5ogfKtZFYV62IKjb3nILkyPhyuyWq9KK9s=; b=flvqHMy+bys9E+OiTvJrTRxuAF 2mwxs5d1F8vUmCLxNQZ/9cbMqk7zRz9Cby15Vo5R9i7IZtRs0SUHY6l+iGkWn1VgRqNfnQs9db9g4 qe2b9C65wcB3c2m6/uIIjKFiY5LA1YzXF95ByqpYgiRJpfvl40t0DjwAgAianPuqjHZPV7u7OSvR1 hCfVNTroR1/xmfGGf83mb1tiwlKwy60J652O8RjWD0BEvr2hggGqlmpvNgq9pkjDT3Rwo0GkVzir1 Ktx1cZ5601QTwrMgy1HqIdjBtd9aSSw5EoWkThK//l9jvC68C6Ic4Q/rxE48AAHf67/w0xUw5U5nk UOdAxlHg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Fa-003YDY-0G; Tue, 12 Sep 2023 14:20:22 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4EM-003XBW-2j for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:17 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-597f461adc5so62618427b3.1 for ; Tue, 12 Sep 2023 07:19:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528345; x=1695133145; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YuFo+zhrlHBQxtpOYSqLrGv0YPox19bH/VMTVW3OMKk=; b=4TLUIqUW1xld3HI23Klmtv2eTGKTX/tuBW8cfMi68O4E4U3Ddmdok9MOTHFNkW3fgd X9gIb+Eiot4GKjRDDU3dTtRI3BuR77noRFRr4ST7j6/i3COjIMNSrRliTBN5yWKZePa+ uZHa3DwglvHxHXiKCnf5G0u3S7VmOVjntvpql1jw4Nm+UqoUTiehQ2Ecuf248WfVYwM1 L+9tVjuFKgOVHi8QmaPi7EPrV06if8I5QJLA/cnQQk1+GgT81MMUltcDp3KdY4ov6zGA Z3bIvTE9SaHf0A/8WGQzO08B2H230rQnoE/+u4i+PdAK0QYrT52kXaSg1l2t2kdw3YzL 83yA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528345; x=1695133145; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YuFo+zhrlHBQxtpOYSqLrGv0YPox19bH/VMTVW3OMKk=; b=iX0fwxD4zUBPp4XUg6M2FLyGDUFApC/7nbcyP8qMcpwcSSEwJ+RMF+tsRdt/ctC1Cg LncU9fM0MWFFfpAMOUKM47TNdWosxPDh7XziLcgi26+W11IB9vsEcKU0npxhZ/QW/BPg FD4c1Ms9BSaG0p2uWRxBaTpxfUILle1c4R1RunKr7jq/U3PBa8bPcAseVmMNWLufpSTZ 7ZoQxfRz0DJaEvxeWS60ySambNdAPgECBTDMCBQEncaUId77/BG+24E5CEIl0EUT7hL2 vYjPm8AdkYhAytBnWU7AFVYw90/FSPLdCfK7fXArYCiH3uiRxOb7YEr/KRI2IRSCo3Ai unLg== X-Gm-Message-State: AOJu0YzRZruuYMBx4L9yQvRzxbPsRfZiFNloF3IoXCmMpTI59TwRotDJ 0lPWzqLVHr17Ucx5JPfXqr1a+cLKNetnhUzJM6cSDDOW5aqii3Uf80PVbQUhtEonSSSnF8T4K+n XQOCfAIi50Ty17q2RJKb3XR39EoX/PGRYCGou4vN0rg3S8elaeKObUEbFRNM8itYNdyRwCW58NC c= X-Google-Smtp-Source: AGHT+IFKQgO4I6kmwuanNWsfDjwN5Tu5o/8KzSYmnaUqgNrAF1UrDmu3cnsKXG71vVUwdOh1zF+Xm5HH X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:c84a:0:b0:584:3d8f:a423 with SMTP id k10-20020a81c84a000000b005843d8fa423mr299933ywl.8.1694528344992; Tue, 12 Sep 2023 07:19:04 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:35 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2581; i=ardb@kernel.org; h=from:subject; bh=V6DZobgx0PLonWJdeZENIUbjbbfoYLLezxmlwMLfJ2U=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6OgBhziF/KqzHAoSsvo5gk3LRBaGX/J/8Gyvvv5Nj 2Vbq1d2lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgImYuTEyPCh7+Ltr6utpnsrv HO/MP6N7z038wrXCZyzZC90LnVWtAhkZ+hM9f4olZXk+f/TuT8NNriqnPzc3sFz3kp7aq6d1Wn8 NKwA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-108-ardb@google.com> Subject: [PATCH v4 45/61] arm64/mm: Avoid #define'ing PTE_MAYBE_NG to 0x0 for asm use From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071906_929662_4F9318E5 X-CRM114-Status: GOOD ( 16.93 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The PROT_* macros resolve to expressions that are only valid in C and not in assembler, and so they are only usable from C code. Currently, we make an exception for the permission indirection init code in proc.S, which doesn't care about the bits that are conditionally set, and so we just #define PTE_MAYBE_NG to 0x0 for any assembler file that includes these definitions. This is dodgy because this means that PROT_NORMAL and friends is generally available in asm code, but defined in a way that deviates from the definition that C code will observe, which might lead to hard to diagnose issues down the road. So instead, #define PTE_MAYBE_NG only in the place where the PIE constants are evaluated, and #undef it again right after. This allows us to drop the #define from pgtable-prot.h, and avoid the risk of deviating definitions between asm and C. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/pgtable-prot.h | 4 ---- arch/arm64/mm/proc.S | 13 +++++++++++++ 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h index eed814b00a38..282e0ba658f0 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -57,10 +57,6 @@ #define _PAGE_READONLY_EXEC (_PAGE_DEFAULT | PTE_USER | PTE_RDONLY | PTE_NG | PTE_PXN) #define _PAGE_EXECONLY (_PAGE_DEFAULT | PTE_RDONLY | PTE_NG | PTE_PXN) -#ifdef __ASSEMBLY__ -#define PTE_MAYBE_NG 0 -#endif - #ifndef __ASSEMBLY__ #include diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 2995a92940e0..cb80ed3a2e52 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -457,11 +457,24 @@ alternative_else_nop_endif ubfx x1, x1, #ID_AA64MMFR3_EL1_S1PIE_SHIFT, #4 cbz x1, .Lskip_indirection + /* + * The PROT_* macros describing the various memory types may resolve to + * C expressions if they include the PTE_MAYBE_* macros, and so they + * can only be used from C code. The PIE_E* constants below are also + * defined in terms of those macros, but will mask out those + * PTE_MAYBE_* constants, whether they are set or not. So #define them + * as 0x0 here so we can evaluate the PIE_E* constants in asm context. + */ + +#define PTE_MAYBE_NG 0 + mov_q x0, PIE_E0 msr REG_PIRE0_EL1, x0 mov_q x0, PIE_E1 msr REG_PIR_EL1, x0 +#undef PTE_MAYBE_NG + mov x0, TCR2_EL1x_PIE msr REG_TCR2_EL1, x0 From patchwork Tue Sep 12 14:16:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 82D02CA0EEB for ; Tue, 12 Sep 2023 14:20:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=v+L0RzQJbSBKnPMO33CFEOuLO5ErEgRqT+Wk8vBSkZI=; b=1cJTs7ibfmklekEBz97XZ6cbeJ q3Tu2ARJ20QAJL9FgWE4JWhBQSYrYPDec7WB+dsS3d55zyeuWIebuxhe5yEZwhFUroTNwuKLv7uZP YyDXCFO5VHbZW84jYPIyyY/E4jx1A3f182OKhcsxWxa5h/t/d7++st5J569oFEAoPGZrYM+jg/MOU 2MgwOMDIWis4EdZkiDZ9TwjJBV4VxTRKEkSZ8ECYEtq2qIqXPP/zVeEhOtbCu7ldWU5D2Fg1DK9hB xeM4LBBUYODTSYHClhVDhSwrTiTLnst97sMUNUx/noPHiQZXQbTuAlRo3/2gHvcAl5ai23gMZ5VKa nvdSSiTg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Fc-003YFY-35; Tue, 12 Sep 2023 14:20:24 +0000 Received: from mail-wm1-f74.google.com ([209.85.128.74]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4ER-003XDe-1i for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:17 +0000 Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-3fef3606d8cso45680575e9.1 for ; Tue, 12 Sep 2023 07:19:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528347; x=1695133147; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=RnjbzTn9sqTG2b9zDcEvyPCMu0VSKd2AbxrHJapCc18=; b=CwrqyFL4a2VQTS5P2bHDTXqFdY6DHOmj4eESChIrXQUmFngd83BHViHJahKRNJKnqA LQhXeqCxSg6wpq5Hb4xQulC9P64AfrAHpQu6MpAn2I0tNs2vuHrp2gbUo78eDRhirHgj w0cuXLz5pULVI3IkOAn+xBo57oQLtDlxmRyyS0DOPmbOkS+El2Fb5fnNpNx9ZkcKqlzL GeYW1YrbniEem4cwp93xCW2FvX4l6SPyCdCHJZDgNSP6Gv4+7r2ZSf8RNruvkFg57VMb G8//mP49OyMBaCX0tIN0hX02kDhc26xXOsZjPis01kf03jh+nkuRZqH7q61LWhzPE1qz oWXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528347; x=1695133147; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RnjbzTn9sqTG2b9zDcEvyPCMu0VSKd2AbxrHJapCc18=; b=Hi8zLjA9mn8aa9PVcOijLa5IceMEPIY8ApJPVrOHjrkRw8CDyLKox4f3vv8NOjSJqE LTCklZWSztsXFlzuBqo/0tTP25JnPUWl+tBl3NVgqghKiDq/GnFIVdwynrZW3zINx3sR eNf3ID+/CU7TCDl9TrbQ2UH/tamnyv1/i0UgqJu05UMtLLHyTID/2H/NJBpwWj3KAgca QRX0aMJLXU2kVQfoa1ruuYibxRei+LguBhigD44eX7DAwmDxG22adpvqkLuQfU6niUdX 9wwmumXC8qVWOEaMAlb03Bh0USUD6V8o8pi2N+PBzOeWBc9rKoW4HvYim5EU4wcCTUXx 2s7A== X-Gm-Message-State: AOJu0YyOCcUtOHmr/hL7wi5++bnPA+OsFvIFtRdWcLv6uHozVzkqSPt+ 1bhiYLBCqQSYLe1UsQDb6LoDdfUyA/GVk/8kCDZ3c83zQFMfK5mEjC879MJ52OROniU5DNJC8xT EVJUaqsSTRkWBDOJUGct82DrxAiOQPy5wAjM7Y/A8+ob4ZrpZvOlNyAsDcriuhGS7z/Pr86Z16f U= X-Google-Smtp-Source: AGHT+IGWWAHJVtdCf3VNXoYWL4zLGB36AztGozpvj7JqDrcpbXwph+/tb0cEE+Oxt3P907usfC1+Oqre X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:2248:b0:3fc:21:2c42 with SMTP id a8-20020a05600c224800b003fc00212c42mr211297wmm.1.1694528347197; Tue, 12 Sep 2023 07:19:07 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:36 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3569; i=ardb@kernel.org; h=from:subject; bh=QFaLenVF+SCYwyd4kXSt0kh1Z++r+4BVATazJPXwzI0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6Jj6xciEqRHtAdrBSw891/l88a32jzVr/W7uZFTW7 /v6gOVMRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZhISC7D/0zLvc9Lb58uCeyQ f5YzTWzZMXZeJsNrCw1jW1ntZh1kDGNk2Gf4qTPzuW3grKgXxb1bd4YG9ZziEfQqupDFd0lQ1PM WKwA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-109-ardb@google.com> Subject: [PATCH v4 46/61] arm64: mm: Wire up TCR.DS bit to PTE shareability fields From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071911_645631_5D76CD32 X-CRM114-Status: GOOD ( 17.16 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel When LPA2 is enabled, bits 8 and 9 of page and block descriptors become part of the output address instead of carrying shareability attributes for the region in question. So avoid setting these bits if TCR.DS == 1, which means LPA2 is enabled. Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 4 ++++ arch/arm64/include/asm/pgtable-prot.h | 18 ++++++++++++++++-- arch/arm64/mm/mmap.c | 4 ++++ arch/arm64/mm/proc.S | 2 ++ 4 files changed, 26 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b10515c0200b..a3f062605fa9 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1344,6 +1344,10 @@ config ARM64_PA_BITS default 48 if ARM64_PA_BITS_48 default 52 if ARM64_PA_BITS_52 +config ARM64_LPA2 + def_bool y + depends on ARM64_PA_BITS_52 && !ARM64_64K_PAGES + choice prompt "Endianness" default CPU_LITTLE_ENDIAN diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h index 282e0ba658f0..c5ec44c42a32 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -30,8 +30,8 @@ #define _PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) #define _PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) -#define PROT_DEFAULT (_PROT_DEFAULT | PTE_MAYBE_NG) -#define PROT_SECT_DEFAULT (_PROT_SECT_DEFAULT | PMD_MAYBE_NG) +#define PROT_DEFAULT (PTE_TYPE_PAGE | PTE_MAYBE_NG | PTE_MAYBE_SHARED | PTE_AF) +#define PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_MAYBE_NG | PMD_MAYBE_SHARED | PMD_SECT_AF) #define PROT_DEVICE_nGnRnE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRnE)) #define PROT_DEVICE_nGnRE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRE)) @@ -67,6 +67,20 @@ extern bool arm64_use_ng_mappings; #define PTE_MAYBE_NG (arm64_use_ng_mappings ? PTE_NG : 0) #define PMD_MAYBE_NG (arm64_use_ng_mappings ? PMD_SECT_NG : 0) +#ifndef CONFIG_ARM64_LPA2 +#define lpa2_is_enabled() false +#define PTE_MAYBE_SHARED PTE_SHARED +#define PMD_MAYBE_SHARED PMD_SECT_S +#else +static inline bool __pure lpa2_is_enabled(void) +{ + return read_tcr() & TCR_DS; +} + +#define PTE_MAYBE_SHARED (lpa2_is_enabled() ? 0 : PTE_SHARED) +#define PMD_MAYBE_SHARED (lpa2_is_enabled() ? 0 : PMD_SECT_S) +#endif + /* * If we have userspace only BTI we don't want to mark kernel pages * guarded even if the system does support BTI. diff --git a/arch/arm64/mm/mmap.c b/arch/arm64/mm/mmap.c index 8f5b7ce857ed..adcf547f74eb 100644 --- a/arch/arm64/mm/mmap.c +++ b/arch/arm64/mm/mmap.c @@ -73,6 +73,10 @@ static int __init adjust_protection_map(void) protection_map[VM_EXEC | VM_SHARED] = PAGE_EXECONLY; } + if (lpa2_is_enabled()) + for (int i = 0; i < ARRAY_SIZE(protection_map); i++) + pgprot_val(protection_map[i]) &= ~PTE_SHARED; + return 0; } arch_initcall(adjust_protection_map); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index cb80ed3a2e52..12305fbefe1f 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -467,6 +467,7 @@ alternative_else_nop_endif */ #define PTE_MAYBE_NG 0 +#define PTE_MAYBE_SHARED 0 mov_q x0, PIE_E0 msr REG_PIRE0_EL1, x0 @@ -474,6 +475,7 @@ alternative_else_nop_endif msr REG_PIR_EL1, x0 #undef PTE_MAYBE_NG +#undef PTE_MAYBE_SHARED mov x0, TCR2_EL1x_PIE msr REG_TCR2_EL1, x0 From patchwork Tue Sep 12 14:16:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381808 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0C0E0CA0EEE for ; Tue, 12 Sep 2023 14:21:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=I0XFTidGzf0zrd5URN20wuQo75JMETZ711xNiQKeifw=; b=xCWKPxk1HnVPGwCvo4Eb5hkmdb oL3N99fLzcB1ik+lVvck+VpupV8rwuj35WKYDiYRoonh0FcWPLf3bil80LfazFStK+9oiUNKmQ0W4 e97j37WwJaIo134wWnFdnqVdZO3xzMlXXAedO7G7fXSni2zv1a7g2Qvx8VZDEfulcummpV8JDxCRP 6yNBQniiqF9rnCRPbQGShgRQU7om2jhZEqibc3xshW9P1rQg3Ay8Ib0qCk6cJRwgT/zp+Q1W1EV/d +JHE8X1NTM6pEun8fOe/+Qj3h8zptfG6T4KR971L81BjgcYueGt/8gn+bh6J1SDIP0z72qslG9Y1c 7axEPHBA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Fe-003YGr-1C; Tue, 12 Sep 2023 14:20:26 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4ER-003XF0-1f for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:20 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-31f46ccee0fso2672502f8f.1 for ; Tue, 12 Sep 2023 07:19:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528349; x=1695133149; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=axKk0zNAoFzdIdM0Ez/IIWZeXThEIBY1gj3iUdXv6TE=; b=A/Qd3XKTbMolDo0nr7XdgawkrtDlcg1+U5TE7fYMwhQi2RY2grym5hOX2DyFhG7iXs NR6fMS0/noHrDzX+OHPzOrbunS7UTMxbptTlVFdn3EJmi/g6SRKWIceYHKwxW3AB5G5m vSAYUPtcMMOejzd3ySNgNe77qdRnHOf6sEm0jNqXVxhQslFg4SHo6cmxwUzTV/NCx1Ns 3k9mlejlJUhSUrpjxU0c2FvrQyAJS3wwCRsFx6X2K7WjVGKC+jvqfW87afl75vpTHzFM /PMv5DtfMsYoj3Ia9ROcNPXmj8J0bXVB38mnP0F7iTAo+GcZBvln/Pu0WgEa2p2FHmQJ o8Gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528349; x=1695133149; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=axKk0zNAoFzdIdM0Ez/IIWZeXThEIBY1gj3iUdXv6TE=; b=GoGT+0MVAq4h+HzD8eIF5vd3upk4YKfeNPoAX2zzioLWXr/IVMCyryr5aP44HQHjEQ +NqGtKdWmpZrl7N3tTjJfjrrY3ZaGzc0BZ0ems28zBpFFanYf48ZUL55xeLeDxeCRfdI lBpFs6ZD47N4lgMn7ducv20CLmFSjTU82N81B8je4mm2HzHueXRjO7IPqzSzvGTViFGC F6y7nneGRLbNiyZAs5hm39aZYC9N/NR+IyDn0WJDlwDUBZ3aZCoRJzhRkKd3yx39uMwN jT0//P8utgNoYvXLC41NQNrlDjd7EaYdrdVtWZSz4Zje5JbSSmKko7snXVFHLegJsiWP Tz5w== X-Gm-Message-State: AOJu0YzWYivQS8YlSWpexwLSI/qxAKln9+PrrS0SofIYi6wlssHarM/G zoXVTDl5rVvlriaV2GTffQ8T3mmSqbJy+s+KpFh9fOlV9GaymiQm//ZePrc4ujC3IVAj8celltY rLU85bkn7l6IxdogOhS9ZXxMTusolLpNykaKcm5Aim4U9Mdu9QnKm+wXLQWwqFhNgrnq9Ykw3+n 8= X-Google-Smtp-Source: AGHT+IHa+Vrf71HijJ8v/ye1eoxXTqJKQIc8+FlI5nhZSuVWo/Z4EJHG7TngdNIO2AhwuuvsamOWKu2v X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:100a:b0:31d:c732:8d4 with SMTP id a10-20020a056000100a00b0031dc73208d4mr53471wrx.3.1694528349354; Tue, 12 Sep 2023 07:19:09 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:37 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=4645; i=ardb@kernel.org; h=from:subject; bh=9BsuuvReeuLNSahj33TapR0erGyCQo/IE359oNWbL7g=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6LjepWbuFW1twiXuJ8Teih3bceSLzQ37B7Ks/mka+ S+Yf2/vKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABPRLGH4nxdWO319y6XqdYVG i21KN4hwpK0PbVlkPduLZS7LzmOX2xkZdh3Yz/TnXfHM7zXXDb90G/E/Nbrg5Hku52yCckqyt78 EDwA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-110-ardb@google.com> Subject: [PATCH v4 47/61] arm64: mm: Add LPA2 support to phys<->pte conversion routines From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071911_682796_BC0825C9 X-CRM114-Status: GOOD ( 16.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In preparation for enabling LPA2 support, introduce the mask values for converting between physical addresses and their representations in a page table descriptor. While at it, move the pte_to_phys asm macro into its only user, so that we can freely modify it to use its input value register as a temp register. For LPA2, the PTE_ADDR_MASK contains two non-adjacent sequences of zero bits, which means it no longer fits into the immediate field of an ordinary ALU instruction. So let's redefine it to include the bits in between as well, and only use it when converting from physical address to PTE representation, where the distinction does not matter. Also update the name accordingly to emphasize this. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 16 ++-------------- arch/arm64/include/asm/pgtable-hwdef.h | 10 +++++++--- arch/arm64/include/asm/pgtable.h | 5 +++-- arch/arm64/mm/proc.S | 8 ++++++++ 4 files changed, 20 insertions(+), 19 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 0710c17800a4..55e8731844cf 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -612,25 +612,13 @@ alternative_endif .macro phys_to_pte, pte, phys #ifdef CONFIG_ARM64_PA_BITS_52 - /* - * We assume \phys is 64K aligned and this is guaranteed by only - * supporting this configuration with 64K pages. - */ - orr \pte, \phys, \phys, lsr #36 - and \pte, \pte, #PTE_ADDR_MASK + orr \pte, \phys, \phys, lsr #PTE_ADDR_HIGH_SHIFT + and \pte, \pte, #PHYS_TO_PTE_ADDR_MASK #else mov \pte, \phys #endif .endm - .macro pte_to_phys, phys, pte - and \phys, \pte, #PTE_ADDR_MASK -#ifdef CONFIG_ARM64_PA_BITS_52 - orr \phys, \phys, \phys, lsl #PTE_ADDR_HIGH_SHIFT - and \phys, \phys, GENMASK_ULL(PHYS_MASK_SHIFT - 1, PAGE_SHIFT) -#endif - .endm - /* * tcr_clear_errata_bits - Clear TCR bits that trigger an errata on this CPU. */ diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index b770f98fc0b5..4426f48f2ae0 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -155,13 +155,17 @@ #define PTE_PXN (_AT(pteval_t, 1) << 53) /* Privileged XN */ #define PTE_UXN (_AT(pteval_t, 1) << 54) /* User XN */ -#define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (48 - PAGE_SHIFT)) - 1) << PAGE_SHIFT) +#define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (50 - PAGE_SHIFT)) - 1) << PAGE_SHIFT) #ifdef CONFIG_ARM64_PA_BITS_52 +#ifdef CONFIG_ARM64_64K_PAGES #define PTE_ADDR_HIGH (_AT(pteval_t, 0xf) << 12) -#define PTE_ADDR_MASK (PTE_ADDR_LOW | PTE_ADDR_HIGH) #define PTE_ADDR_HIGH_SHIFT 36 +#define PHYS_TO_PTE_ADDR_MASK (PTE_ADDR_LOW | PTE_ADDR_HIGH) #else -#define PTE_ADDR_MASK PTE_ADDR_LOW +#define PTE_ADDR_HIGH (_AT(pteval_t, 0x3) << 8) +#define PTE_ADDR_HIGH_SHIFT 42 +#define PHYS_TO_PTE_ADDR_MASK GENMASK_ULL(49, 8) +#endif #endif /* diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 1858abd51b8f..2d1294b532f4 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -80,15 +80,16 @@ extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)]; #ifdef CONFIG_ARM64_PA_BITS_52 static inline phys_addr_t __pte_to_phys(pte_t pte) { + pte_val(pte) &= ~PTE_MAYBE_SHARED; return (pte_val(pte) & PTE_ADDR_LOW) | ((pte_val(pte) & PTE_ADDR_HIGH) << PTE_ADDR_HIGH_SHIFT); } static inline pteval_t __phys_to_pte_val(phys_addr_t phys) { - return (phys | (phys >> PTE_ADDR_HIGH_SHIFT)) & PTE_ADDR_MASK; + return (phys | (phys >> PTE_ADDR_HIGH_SHIFT)) & PHYS_TO_PTE_ADDR_MASK; } #else -#define __pte_to_phys(pte) (pte_val(pte) & PTE_ADDR_MASK) +#define __pte_to_phys(pte) (pte_val(pte) & PTE_ADDR_LOW) #define __phys_to_pte_val(phys) (phys) #endif diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 12305fbefe1f..31be0d16241d 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -205,6 +205,14 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .pushsection ".idmap.text", "a" + .macro pte_to_phys, phys, pte + and \phys, \pte, #PTE_ADDR_LOW +#ifdef CONFIG_ARM64_PA_BITS_52 + and \pte, \pte, #PTE_ADDR_HIGH + orr \phys, \phys, \pte, lsl #PTE_ADDR_HIGH_SHIFT +#endif + .endm + .macro kpti_mk_tbl_ng, type, num_entries add end_\type\()p, cur_\type\()p, #\num_entries * 8 .Ldo_\type: From patchwork Tue Sep 12 14:16:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 73B93CA0EED for ; Tue, 12 Sep 2023 14:21:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=IHUUiv38Jn/nmMyAT5VfE7ebxeS+XQ5H347m/9itFao=; b=E0W8CAeKlRNnnK86MoiBGWc5Ww 6fBrFE0EKEQdgBREiDAwMeUoLwRumuo20dKrm2IMuryUZIS2UPXUNzk9E5MsQ85UDG1ZVzlvB7mvm 8cAE3k9klGfslOtpAo/hKTb/PzVQaHEjxNas/z61ykFzBdHExCD9ZjWFIgVZtmolEo+fdzwnWvI3l /jvRrW5qisVjWcWuaBwm6doJ5FpdNahvv0DRoxC8YSq20Ho6rMQIExHlxnfH0+BnwdvrMg2OB2INh S+ZevzBSAQ7SLIc13FvuUVoAGhEcCjID6ZIWdOnlN77ycMhNDkAlVc+L2EtYkGdOUfcJKLjLVFdTW tko+/Hhg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Fi-003YJU-0p; Tue, 12 Sep 2023 14:20:30 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4ET-003XHC-22 for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:23 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-402c46c4a04so43039985e9.2 for ; Tue, 12 Sep 2023 07:19:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528351; x=1695133151; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3bHaxkXzhxvGKFXmJcqS1wtte4sDGQRztFvukt6wPec=; b=FIDhwLUP/yfdIbSjvWCUgTkHf1HLRJlPHSe2lHj84sdcC/knmKw3geuqd4XSxx/wTe KawRtzQbTxedKZBCFXYbj6lEktmF8r54lZNvWYD1849ZV+ys0s5RvVccqfU7yIu6Wknp R7zTC4kwC8F81UnJMC9AmtZE+JZc0djg0/2YIUN10Fw2kQRCy+Pm4F4BH/JuhJOE1zbx j58KFEmUbzvjNkS7DHxESmfy7Rfch1FmyqZw7mpQ+/LJF/qyCSk+dsVYoljzdAx+Vb5m cKtOZt7jCB1Yr/9zzSLHL9jGHvOxv2kmcPpXuKROc9E9w/tIBMmN65aLujoTyCX+IMfJ 6dBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528351; x=1695133151; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3bHaxkXzhxvGKFXmJcqS1wtte4sDGQRztFvukt6wPec=; b=rBoD6UOSCC2NZbaYFV+cnyA4GxAIka3i9hcUF2VGBisreiMdEhtLEg7j7mcHWt4XO3 R21U12cRd6hhvE1/HXdrDVGs+WprZQxQ6wFtrXP2jAcS/aKUeMOWRLMT7y2XDJ09aA0b Xh8OO3h0uaq0WXOqefibYKT0i1PB2FOoEAa5KQAh+2mWS/ikIS6UuzIi9BO8SX+pIySZ OYrCCrY10RkhQOgtG0c5xp+oK8DRvruodk/qR47wPs+7yuKb9CCOjKi8qv+62tLqX6sg uikrvLFU6vb90oXKbsxzFcdteAhbboZ4E1pvEL6u/55Jr8MDtlgHy85dng5hJYN4zjH1 YmrA== X-Gm-Message-State: AOJu0Yyl8CfmXvdcsbWhr839pkNLrPjsPLZ7QjNO7/PWU0V3ji8FXcmK 18rBzut6KHZ+JxOir208P+8957ChXdODU/SOztUOvo3ZVH3uB5LZ+13nSIWd0aPg3e2J785lxSY BGTEIn6Xtc/ZTSJiL35xLVUkT7Rwnu1Mq5DMiaXGua1PbqCPLClzkFoCKey3rWHIl9gw/13fXNg c= X-Google-Smtp-Source: AGHT+IGyCmJND6uGBkZVwgfA2p7zjdcoTsxMlEZq02sUqAmPpEaHHqXlsbTyPBKGWO1y3Ntibvl/KIZT X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a1c:7914:0:b0:402:eac3:7a5e with SMTP id l20-20020a1c7914000000b00402eac37a5emr216564wme.3.1694528351550; Tue, 12 Sep 2023 07:19:11 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:38 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10782; i=ardb@kernel.org; h=from:subject; bh=uGTAqqb+ogGidSNUTuAXAW/9slzvBPb0KbBnx4xxiPQ=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6ISpBkvD0yccejOmrk/bd7Qx8bLob9v6socH2vefv xS2b2FpRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjITx1Ghq/prYczKqbfea/Y klCpeZ5lxyeWVU5J3NLr2efN+aj9jYnhN1vzmj+p24rm33XJyis3vRgw4bqaFnOsnWOVWJbZn2f lLAA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-111-ardb@google.com> Subject: [PATCH v4 48/61] arm64: mm: Add definitions to support 5 levels of paging From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071913_731986_CE24DB1E X-CRM114-Status: GOOD ( 24.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add the required types and descriptor accessors to support 5 levels of paging in the common code. This is one of the prerequisites for supporting 52-bit virtual addressing with 4k pages. Note that this does not cover the code that handles kernel mappings or the fixmap. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/pgalloc.h | 41 ++++++++++ arch/arm64/include/asm/pgtable-hwdef.h | 22 +++++- arch/arm64/include/asm/pgtable-types.h | 6 ++ arch/arm64/include/asm/pgtable.h | 82 +++++++++++++++++++- arch/arm64/mm/mmu.c | 31 +++++++- arch/arm64/mm/pgd.c | 15 +++- 6 files changed, 188 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 237224484d0f..cae8c648f462 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -60,6 +60,47 @@ static inline void __p4d_populate(p4d_t *p4dp, phys_addr_t pudp, p4dval_t prot) } #endif /* CONFIG_PGTABLE_LEVELS > 3 */ +#if CONFIG_PGTABLE_LEVELS > 4 + +static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t prot) +{ + if (pgtable_l5_enabled()) + set_pgd(pgdp, __pgd(__phys_to_pgd_val(p4dp) | prot)); +} + +static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) +{ + pgdval_t pgdval = PGD_TYPE_TABLE; + + pgdval |= (mm == &init_mm) ? PGD_TABLE_UXN : PGD_TABLE_PXN; + __pgd_populate(pgdp, __pa(p4dp), pgdval); +} + +static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) +{ + gfp_t gfp = GFP_PGTABLE_USER; + + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + return (p4d_t *)get_zeroed_page(gfp); +} + +static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + if (!pgtable_l5_enabled()) + return; + BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + free_page((unsigned long)p4d); +} + +#define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) +#else +static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t prot) +{ + BUILD_BUG(); +} +#endif /* CONFIG_PGTABLE_LEVELS > 4 */ + extern pgd_t *pgd_alloc(struct mm_struct *mm); extern void pgd_free(struct mm_struct *mm, pgd_t *pgdp); diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index 4426f48f2ae0..ef207a0d4f0d 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -26,10 +26,10 @@ #define ARM64_HW_PGTABLE_LEVELS(va_bits) (((va_bits) - 4) / (PAGE_SHIFT - 3)) /* - * Size mapped by an entry at level n ( 0 <= n <= 3) + * Size mapped by an entry at level n ( -1 <= n <= 3) * We map (PAGE_SHIFT - 3) at all translation levels and PAGE_SHIFT bits * in the final page. The maximum number of translation levels supported by - * the architecture is 4. Hence, starting at level n, we have further + * the architecture is 5. Hence, starting at level n, we have further * ((4 - n) - 1) levels of translation excluding the offset within the page. * So, the total number of bits mapped by an entry at level n is : * @@ -62,9 +62,16 @@ #define PTRS_PER_PUD (1 << (PAGE_SHIFT - 3)) #endif +#if CONFIG_PGTABLE_LEVELS > 4 +#define P4D_SHIFT ARM64_HW_PGTABLE_LEVEL_SHIFT(0) +#define P4D_SIZE (_AC(1, UL) << P4D_SHIFT) +#define P4D_MASK (~(P4D_SIZE-1)) +#define PTRS_PER_P4D (1 << (PAGE_SHIFT - 3)) +#endif + /* * PGDIR_SHIFT determines the size a top-level page table entry can map - * (depending on the configuration, this level can be 0, 1 or 2). + * (depending on the configuration, this level can be -1, 0, 1 or 2). */ #define PGDIR_SHIFT ARM64_HW_PGTABLE_LEVEL_SHIFT(4 - CONFIG_PGTABLE_LEVELS) #define PGDIR_SIZE (_AC(1, UL) << PGDIR_SHIFT) @@ -87,6 +94,15 @@ /* * Hardware page table definitions. * + * Level -1 descriptor (PGD). + */ +#define PGD_TYPE_TABLE (_AT(pgdval_t, 3) << 0) +#define PGD_TABLE_BIT (_AT(pgdval_t, 1) << 1) +#define PGD_TYPE_MASK (_AT(pgdval_t, 3) << 0) +#define PGD_TABLE_PXN (_AT(pgdval_t, 1) << 59) +#define PGD_TABLE_UXN (_AT(pgdval_t, 1) << 60) + +/* * Level 0 descriptor (P4D). */ #define P4D_TYPE_TABLE (_AT(p4dval_t, 3) << 0) diff --git a/arch/arm64/include/asm/pgtable-types.h b/arch/arm64/include/asm/pgtable-types.h index b8f158ae2527..6d6d4065b0cb 100644 --- a/arch/arm64/include/asm/pgtable-types.h +++ b/arch/arm64/include/asm/pgtable-types.h @@ -36,6 +36,12 @@ typedef struct { pudval_t pud; } pud_t; #define __pud(x) ((pud_t) { (x) } ) #endif +#if CONFIG_PGTABLE_LEVELS > 4 +typedef struct { p4dval_t p4d; } p4d_t; +#define p4d_val(x) ((x).p4d) +#define __p4d(x) ((p4d_t) { (x) } ) +#endif + typedef struct { pgdval_t pgd; } pgd_t; #define pgd_val(x) ((x).pgd) #define __pgd(x) ((pgd_t) { (x) } ) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 2d1294b532f4..769abda8a4f9 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -800,7 +800,6 @@ static inline pud_t *p4d_pgtable(p4d_t p4d) #else #define p4d_page_paddr(p4d) ({ BUILD_BUG(); 0;}) -#define pgd_page_paddr(pgd) ({ BUILD_BUG(); 0;}) /* Match pud_offset folding in */ #define pud_set_fixmap(addr) NULL @@ -811,6 +810,87 @@ static inline pud_t *p4d_pgtable(p4d_t p4d) #endif /* CONFIG_PGTABLE_LEVELS > 3 */ +#if CONFIG_PGTABLE_LEVELS > 4 + +static __always_inline bool pgtable_l5_enabled(void) +{ + if (!alternative_has_cap_likely(ARM64_ALWAYS_BOOT)) + return vabits_actual == VA_BITS; + return alternative_has_cap_unlikely(ARM64_HAS_VA52); +} + +static inline bool mm_p4d_folded(const struct mm_struct *mm) +{ + return !pgtable_l5_enabled(); +} +#define mm_p4d_folded mm_p4d_folded + +#define p4d_ERROR(e) \ + pr_err("%s:%d: bad p4d %016llx.\n", __FILE__, __LINE__, p4d_val(e)) + +#define pgd_none(pgd) (pgtable_l5_enabled() && !pgd_val(pgd)) +#define pgd_bad(pgd) (pgtable_l5_enabled() && !(pgd_val(pgd) & 2)) +#define pgd_present(pgd) (!pgd_none(pgd)) + +static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) +{ + if (in_swapper_pgdir(pgdp)) { + set_swapper_pgd(pgdp, __pgd(pgd_val(pgd))); + return; + } + + WRITE_ONCE(*pgdp, pgd); + dsb(ishst); + isb(); +} + +static inline void pgd_clear(pgd_t *pgdp) +{ + if (pgtable_l5_enabled()) + set_pgd(pgdp, __pgd(0)); +} + +static inline phys_addr_t pgd_page_paddr(pgd_t pgd) +{ + return __pgd_to_phys(pgd); +} + +#define p4d_index(addr) (((addr) >> P4D_SHIFT) & (PTRS_PER_P4D - 1)) + +static inline p4d_t *pgd_to_folded_p4d(pgd_t *pgdp, unsigned long addr) +{ + return (p4d_t *)PTR_ALIGN_DOWN(pgdp, PAGE_SIZE) + p4d_index(addr); +} + +static inline phys_addr_t p4d_offset_phys(pgd_t *pgdp, unsigned long addr) +{ + BUG_ON(!pgtable_l5_enabled()); + + return pgd_page_paddr(READ_ONCE(*pgdp)) + p4d_index(addr) * sizeof(p4d_t); +} + +static inline +p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd, unsigned long addr) +{ + if (!pgtable_l5_enabled()) + return pgd_to_folded_p4d(pgdp, addr); + return (p4d_t *)__va(pgd_page_paddr(pgd)) + p4d_index(addr); +} +#define p4d_offset_lockless p4d_offset_lockless + +static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long addr) +{ + return p4d_offset_lockless(pgdp, READ_ONCE(*pgdp), addr); +} + +#define pgd_page(pgd) pfn_to_page(__phys_to_pfn(__pgd_to_phys(pgd))) + +#else + +static inline bool pgtable_l5_enabled(void) { return false; } + +#endif /* CONFIG_PGTABLE_LEVELS > 4 */ + #define pgd_ERROR(e) \ pr_err("%s:%d: bad pgd %016llx.\n", __FILE__, __LINE__, pgd_val(e)) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 50972763a67e..2212289c9a02 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1022,7 +1022,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr, if (CONFIG_PGTABLE_LEVELS <= 3) return; - if (!pgtable_range_aligned(start, end, floor, ceiling, PGDIR_MASK)) + if (!pgtable_range_aligned(start, end, floor, ceiling, P4D_MASK)) return; /* @@ -1045,8 +1045,8 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr, unsigned long end, unsigned long floor, unsigned long ceiling) { - unsigned long next; p4d_t *p4dp, p4d; + unsigned long i, next, start = addr; do { next = p4d_addr_end(addr, end); @@ -1058,6 +1058,27 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr, WARN_ON(!p4d_present(p4d)); free_empty_pud_table(p4dp, addr, next, floor, ceiling); } while (addr = next, addr < end); + + if (!pgtable_l5_enabled()) + return; + + if (!pgtable_range_aligned(start, end, floor, ceiling, PGDIR_MASK)) + return; + + /* + * Check whether we can free the p4d page if the rest of the + * entries are empty. Overlap with other regions have been + * handled by the floor/ceiling check. + */ + p4dp = p4d_offset(pgdp, 0UL); + for (i = 0; i < PTRS_PER_P4D; i++) { + if (!p4d_none(READ_ONCE(p4dp[i]))) + return; + } + + pgd_clear(pgdp); + __flush_tlb_kernel_pgtable(start); + free_hotplug_pgtable_page(virt_to_page(p4dp)); } static void free_empty_tables(unsigned long addr, unsigned long end, @@ -1142,6 +1163,12 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t prot) return 1; } +#ifndef __PAGETABLE_P4D_FOLDED +void p4d_clear_huge(p4d_t *p4dp) +{ +} +#endif + int pud_clear_huge(pud_t *pudp) { if (!pud_sect(READ_ONCE(*pudp))) diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c index 4a64089e5771..3c4f8a279d2b 100644 --- a/arch/arm64/mm/pgd.c +++ b/arch/arm64/mm/pgd.c @@ -17,11 +17,20 @@ static struct kmem_cache *pgd_cache __ro_after_init; +static bool pgdir_is_page_size(void) +{ + if (PGD_SIZE == PAGE_SIZE) + return true; + if (CONFIG_PGTABLE_LEVELS == 5) + return !pgtable_l5_enabled(); + return false; +} + pgd_t *pgd_alloc(struct mm_struct *mm) { gfp_t gfp = GFP_PGTABLE_USER; - if (PGD_SIZE == PAGE_SIZE) + if (pgdir_is_page_size()) return (pgd_t *)__get_free_page(gfp); else return kmem_cache_alloc(pgd_cache, gfp); @@ -29,7 +38,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm) void pgd_free(struct mm_struct *mm, pgd_t *pgd) { - if (PGD_SIZE == PAGE_SIZE) + if (pgdir_is_page_size()) free_page((unsigned long)pgd); else kmem_cache_free(pgd_cache, pgd); @@ -37,7 +46,7 @@ void pgd_free(struct mm_struct *mm, pgd_t *pgd) void __init pgtable_cache_init(void) { - if (PGD_SIZE == PAGE_SIZE) + if (pgdir_is_page_size()) return; #ifdef CONFIG_ARM64_PA_BITS_52 From patchwork Tue Sep 12 14:16:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381809 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2538FCA0EEB for ; Tue, 12 Sep 2023 14:21:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Ds+WE+f3pm319hiZlXLLmMZrVcR2lACzPbKIbUkKchU=; b=FmnBul6nvyrONJBqMV+yTLKJpa gEasm8EHCawmDkkyDNvO/Kms7kkz326+pQcWQgT5TfJi4PAZAD/V5bMlEzYwAHlIxAdMKe6ZkzJdK WQ5SZKtXIfDseo6GfbGYtk8ci3oWdchr69ysCYVqmWHXJa39f5dAphVpuWpztzeYfbDqt6S/9XwTW w1UpvB7gw9W/bIN5KrNmuWLwuRNcMSgnolGfRWXNVbZxYcRAlcvzl2SYZcVifWmxNgmkoOJ/4a7mr 28re9WD85TUr8vbAnqRDQvvOBbT7K2ROpYQcf/IzrH8KNaprVUcCsMvRkV2OZkhby1u6fM0nBfr+k UV2xO7bA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Fl-003YMn-1x; Tue, 12 Sep 2023 14:20:33 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4EW-003XJw-19 for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:25 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-58e49935630so104552307b3.0 for ; Tue, 12 Sep 2023 07:19:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528354; x=1695133154; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7BFmaSziYEEiu5DiUfOMF9ycyq5S9ydLnoCOi72SCs4=; b=g8crgThcKeqTr1Ozb3XpPnhAcU3AjbcmDTnjzCNSbd7gRJfAt+SYGM3sjGLcPOa5BK ralQbeKKqcsDOq5wMJXORvZ/LNhpIaG1vCZPhFOUCRwm5/n6y7Ap2m+X7Cr0Jy3QJs6s sOQmDGSg2eYPm/MYJK9mu1jnnhwxNBUAu7a/JegnSRM9mG9C4nwKfhu4u9aXeOCkpTfx fepiey3O/jfRZK+12iEhnyNBqvgFweqRD3lQB9o7Ajr118XH00K2670oZx07RuWelSOz 4BTSEmVaji81ih5ZUg0NyFNy4pqts5G/V/YbSvqnVb34GezRsb9vyQSJjVLhG8eqIdS9 h5yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528354; x=1695133154; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7BFmaSziYEEiu5DiUfOMF9ycyq5S9ydLnoCOi72SCs4=; b=tO81Toz2R8rmrWHsZcsL7MURntZV0FRiLN04hX59G2+0znZF2bUpLtH4vlrUNTH/ZJ vohvUMS8qAiJE9zjBFExkBAauT8eMiiyJCQJZ02LVLe9hIPtq8qAYOATIUcWDAcTUrXJ EwsM7hzFu+Ny3QLEoKm8PcSQ+FWC8L3qw3cVMJ7lcZhltFjjf9qOzXYQAMKQV/VcOgSK YUt7Vft7F/YhBTHL4eCNF+jY+fgxYgE+9j5HOGMlY7sDQDC4ifXe2ZYoksEMJpHHy20P WXvwiiMCgSHlvdcauDNPhyuve3slC/1EN1kWC8QfM76ofOqBHhbQCHOfxhOnSRJEO1GX F/rw== X-Gm-Message-State: AOJu0YyA3DFQWXb5T+YWQrVIMXQH2eIyB381E8hZ/tlAIl84tKnKA4Pw IHCmj10lxQnGnyAu343qUx1Il8iP5wMY0n6QUwjAI9VZ4Ad+SNL3mSuwmLtStyLjVM0+KXBRIxr +nxxhjjgRmoezqS0QMCCrdVgpeCweCIP0PV0k6bgVCD94hrp9OASd5TD4sXvA6ZFuAiukM+KTle U= X-Google-Smtp-Source: AGHT+IFDXWvktD9Qvfuk4AAck/vdyq3TLq/UH3KTSPcR6uU3mesB/qhDLqanJ3ZGfEEBbxYe2FEkRoVW X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:690c:2d13:b0:589:a533:405b with SMTP id eq19-20020a05690c2d1300b00589a533405bmr63505ywb.3.1694528354354; Tue, 12 Sep 2023 07:19:14 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:39 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6547; i=ardb@kernel.org; h=from:subject; bh=qXSALkyQaGwy5loMod8vAXTsYLFRxnid2WwT8yST+cU=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6ORZjZqMHYamfFNyn949+Kjnl+jrT73BEtsuGHD1P lY13WzVUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACZy7Tojw8zqtK072694/fCT 73W9ZBH0IcftxqRP4VzZKafW9l239WX4ny0kPfv0ug0TMjYFvHJsuJN08/EKjU2mz051L3as3CO nygUA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-112-ardb@google.com> Subject: [PATCH v4 49/61] arm64: mm: add LPA2 and 5 level paging support to G-to-nG conversion From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071916_439638_EB8A019A X-CRM114-Status: GOOD ( 21.28 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add support for 5 level paging in the G-to-nG routine that creates its own temporary page tables to traverse the swapper page tables. Also add support for running the 5 level configuration with the top level folded at runtime, to support CPUs that do not implement the LPA2 extension. While at it, wire up the level skipping logic so it will also trigger on 4 level configurations with LPA2 enabled at build time but not active at runtime, as we'll fall back to 3 level paging in that case. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/cpufeature.c | 9 ++- arch/arm64/mm/proc.S | 70 +++++++++++++++++--- 2 files changed, 66 insertions(+), 13 deletions(-) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index a2c320ed36b3..f4c81300cb13 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1753,6 +1753,9 @@ kpti_install_ng_mappings(const struct arm64_cpu_capabilities *__unused) if (arm64_use_ng_mappings) return; + if (levels == 5 && !pgtable_l5_enabled()) + levels = 4; + remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings); if (!cpu) { @@ -1766,9 +1769,9 @@ kpti_install_ng_mappings(const struct arm64_cpu_capabilities *__unused) // // The physical pages are laid out as follows: // - // +--------+-/-------+-/------ +-\\--------+ - // : PTE[] : | PMD[] : | PUD[] : || PGD[] : - // +--------+-\-------+-\------ +-//--------+ + // +--------+-/-------+-/------ +-/------ +-\\\--------+ + // : PTE[] : | PMD[] : | PUD[] : | P4D[] : ||| PGD[] : + // +--------+-\-------+-\------ +-\------ +-///--------+ // ^ // The first page is mapped into this hierarchy at a PMD_SHIFT // aligned virtual address, so that we can manipulate the PTE diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 31be0d16241d..d60366144031 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -216,16 +216,15 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .macro kpti_mk_tbl_ng, type, num_entries add end_\type\()p, cur_\type\()p, #\num_entries * 8 .Ldo_\type: - ldr \type, [cur_\type\()p] // Load the entry + ldr \type, [cur_\type\()p], #8 // Load the entry and advance tbz \type, #0, .Lnext_\type // Skip invalid and tbnz \type, #11, .Lnext_\type // non-global entries orr \type, \type, #PTE_NG // Same bit for blocks and pages - str \type, [cur_\type\()p] // Update the entry + str \type, [cur_\type\()p, #-8] // Update the entry .ifnc \type, pte tbnz \type, #1, .Lderef_\type .endif .Lnext_\type: - add cur_\type\()p, cur_\type\()p, #8 cmp cur_\type\()p, end_\type\()p b.ne .Ldo_\type .endm @@ -235,18 +234,18 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) * fixmap slot associated with the current level. */ .macro kpti_map_pgtbl, type, level - str xzr, [temp_pte, #8 * (\level + 1)] // break before make + str xzr, [temp_pte, #8 * (\level + 2)] // break before make dsb nshst - add pte, temp_pte, #PAGE_SIZE * (\level + 1) + add pte, temp_pte, #PAGE_SIZE * (\level + 2) lsr pte, pte, #12 tlbi vaae1, pte dsb nsh isb phys_to_pte pte, cur_\type\()p - add cur_\type\()p, temp_pte, #PAGE_SIZE * (\level + 1) + add cur_\type\()p, temp_pte, #PAGE_SIZE * (\level + 2) orr pte, pte, pte_flags - str pte, [temp_pte, #8 * (\level + 1)] + str pte, [temp_pte, #8 * (\level + 2)] dsb nshst .endm @@ -279,6 +278,8 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) end_ptep .req x15 pte .req x16 valid .req x17 + cur_p4dp .req x19 + end_p4dp .req x20 mov x5, x3 // preserve temp_pte arg mrs swapper_ttb, ttbr1_el1 @@ -286,6 +287,12 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) cbnz cpu, __idmap_kpti_secondary +#if CONFIG_PGTABLE_LEVELS > 4 + stp x29, x30, [sp, #-32]! + mov x29, sp + stp x19, x20, [sp, #16] +#endif + /* We're the boot CPU. Wait for the others to catch up */ sevl 1: wfe @@ -303,9 +310,32 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) mov_q pte_flags, KPTI_NG_PTE_FLAGS /* Everybody is enjoying the idmap, so we can rewrite swapper. */ + +#ifdef CONFIG_ARM64_LPA2 + /* + * If LPA2 support is configured, but 52-bit virtual addressing is not + * enabled at runtime, we will fall back to one level of paging less, + * and so we have to walk swapper_pg_dir as if we dereferenced its + * address from a PGD level entry, and terminate the PGD level loop + * right after. + */ + adrp pgd, swapper_pg_dir // walk &swapper_pg_dir at the next level + mov cur_pgdp, end_pgdp // must be equal to terminate the PGD loop +alternative_if_not ARM64_HAS_VA52 + b .Lderef_pgd // skip to the next level +alternative_else_nop_endif + /* + * LPA2 based 52-bit virtual addressing requires 52-bit physical + * addressing to be enabled as well. In this case, the shareability + * bits are repurposed as physical address bits, and should not be + * set in pte_flags. + */ + bic pte_flags, pte_flags, #PTE_SHARED +#endif + /* PGD */ adrp cur_pgdp, swapper_pg_dir - kpti_map_pgtbl pgd, 0 + kpti_map_pgtbl pgd, -1 kpti_mk_tbl_ng pgd, PTRS_PER_PGD /* Ensure all the updated entries are visible to secondary CPUs */ @@ -318,16 +348,33 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) /* Set the flag to zero to indicate that we're all done */ str wzr, [flag_ptr] +#if CONFIG_PGTABLE_LEVELS > 4 + ldp x19, x20, [sp, #16] + ldp x29, x30, [sp], #32 +#endif ret .Lderef_pgd: + /* P4D */ + .if CONFIG_PGTABLE_LEVELS > 4 + p4d .req x30 + pte_to_phys cur_p4dp, pgd + kpti_map_pgtbl p4d, 0 + kpti_mk_tbl_ng p4d, PTRS_PER_P4D + b .Lnext_pgd + .else /* CONFIG_PGTABLE_LEVELS <= 4 */ + p4d .req pgd + .set .Lnext_p4d, .Lnext_pgd + .endif + +.Lderef_p4d: /* PUD */ .if CONFIG_PGTABLE_LEVELS > 3 pud .req x10 - pte_to_phys cur_pudp, pgd + pte_to_phys cur_pudp, p4d kpti_map_pgtbl pud, 1 kpti_mk_tbl_ng pud, PTRS_PER_PUD - b .Lnext_pgd + b .Lnext_p4d .else /* CONFIG_PGTABLE_LEVELS <= 3 */ pud .req pgd .set .Lnext_pud, .Lnext_pgd @@ -371,6 +418,9 @@ SYM_TYPED_FUNC_START(idmap_kpti_install_ng_mappings) .unreq end_ptep .unreq pte .unreq valid + .unreq cur_p4dp + .unreq end_p4dp + .unreq p4d /* Secondary CPUs end up here */ __idmap_kpti_secondary: From patchwork Tue Sep 12 14:16:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381866 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AF183CA0EF4 for ; Tue, 12 Sep 2023 15:30:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=foZ2q1NTIr4JKQ1c11WeGQ1AVrD5XC/pLfjPgaHfIUw=; b=p/KCwgxw7J3UBAQ78YlYVXgJ08 F/v1nF6+wrDFIA+ovcrnCvRkU9YODRGqhpw12nJ6Bhq54Ru9YxFKgUmXebDg8YpwIdBk7jDEDwhZz Jfb6ibqDtVKUGC3+1wTmiYohUDDHUMlZ4q8uv8LQV0ajak112y4TBY9ovpZ6DktUKZ41bF91Bhp84 RICTL1VHsE4LeXRri8LJi8j1bF0i0cyXkznDNExCBT1b/vHxf1/aFCdm1lPD+yCtocm5PkunyX2iY 8+9Wa/K0PwI3gd9ProNcI2nvq2ohNZ9ypNPtksouUCBypnsLmAAsGYWjWWJD/RFj5e9ggJkslF71Y p48YkHfA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg5LR-003ihM-0r; Tue, 12 Sep 2023 15:30:29 +0000 Received: from mail-wm1-f74.google.com ([209.85.128.74]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4EZ-003XMd-0n for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:29 +0000 Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-402493d2997so41148955e9.0 for ; Tue, 12 Sep 2023 07:19:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528357; x=1695133157; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6kpHMh1fxJX0WosZvnbK8dUUkz2qB4AwpNaVnaT03lM=; b=CTEPN2PKwmuXw14K43zldkyz5xCD7Lf+pDc9sB9zBlrzjyyq/5MQBmTKTQFUnTzfvU 4Puva8sJjIX6isBrevZYhwMlswDWAOAaJuQckQk4pZKXz8WqIcscJ/QYPPUnpLmw3/0S K1zbW9PyEKuotrBhUHYA8SIsVktbG4lZlWHyf2xvJlj0j2o7I9FL82iZwwpVp0+T3Rpe GrzNp1xa/lgs1jXWg+Fb4zrjHi9oEXOkX59lbTrPgWExrBkSy7psuSgdEYygVn4HpAC8 BpIw2RlbQMDSksaouik3kL/wM352AGD/0aeROv06+LSTJKJR3EplH8/ylAyt+WH5ToId ycsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528357; x=1695133157; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6kpHMh1fxJX0WosZvnbK8dUUkz2qB4AwpNaVnaT03lM=; b=InFr6nBrcUHjbjffsG+dyWTXhLedzg1Bne+bq1GMc7fXKyJJNiDFlqfQvdPcadqrB3 ombOtwNdSa8V0WOfOtUEx0/QrIxGcG+uBwIjlFMqaalg6HsnQwCV0R8/SiVB9XQ4x0dP gaK+ui6ShlufF/bKpB4TzsZqvIDHmeEsRERPYsgoZ9GJpfGD8HW/FKGhP4jm2o9JeD0c 9YaleZ18/8sOX4jR74DPgLNtGs3K/wHTEy0HYNuRcj8X0Mjdhh1iZM9E1/bvfZSKgkIP CCCEQpWYfVf5YdMIc3dKsbhlq/m/o0vcrzRIcdytv/2tDmW0ASx7Fst5wGWJuhdOqTuJ GXag== X-Gm-Message-State: AOJu0YyvqvIa2djIZNafrOFP0w4vUp9LG8LHP0iY7N3fWidSmfw1cbyM BD4+1HR9PqnGjiedtZLPzJR9DVySyBqJFXCt9nQOW9GScnC2BXaWrChCY44ZvnxsovwFXCu6mhs aFXJm+mOMNulGFslbb77XVok0aoQzghYx6KYyDcSldQ2cMT2eTLiPB3ZC68ENOtRYTkWbh+mBBM Q= X-Google-Smtp-Source: AGHT+IHmr98bQ0q5U401wQcQr4IP3HtQdtRQM1fdsEnvbniJkta9Di2RPLFxXmznY6Ni5GeK2Vcb6cT0 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a7b:cd96:0:b0:3fb:bcec:72da with SMTP id y22-20020a7bcd96000000b003fbbcec72damr214342wmj.5.1694528356893; Tue, 12 Sep 2023 07:19:16 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:40 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=13274; i=ardb@kernel.org; h=from:subject; bh=KJROpUVFPieDH4rah4e246WaYyoYeAT8rn8KGExAweE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6NTr11mVF+csOZn2pKLsRKrHBSm9hRcff2HkVgkXO vBheu7KjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRvasYGb6U8IgG6mtuuOGy ho/z2LqyB4xlgTN4zinq8Mj8Xlybtojhr8CMzZJB+Uu7Q/q3fHw2v2vHtI9beByZP1/Pm7do0yW lBD4A X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-113-ardb@google.com> Subject: [PATCH v4 50/61] arm64: Enable LPA2 at boot if supported by the system From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071919_326921_3C6CD054 X-CRM114-Status: GOOD ( 33.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Update the early kernel mapping code to take 52-bit virtual addressing into account based on the LPA2 feature. This is a bit more involved than LVA (which is supported with 64k pages only), given that some page table descriptor bits change meaning in this case. To keep the handling in asm to a minimum, the initial ID map is still created with 48-bit virtual addressing, which implies that the kernel image must be loaded into 48-bit addressable physical memory. This is currently required by the boot protocol, even though we happen to support placement outside of that for LVA/64k based configurations. Enabling LPA2 involves more than setting TCR.T1SZ to a lower value, there is also a DS bit in TCR that needs to be set, and which changes the meaning of bits [9:8] in all page table descriptors. Since we cannot enable DS and every live page table descriptor at the same time, let's pivot through another temporary mapping. This avoids the need to reintroduce manipulations of the page tables with the MMU and caches disabled. To permit the LPA2 feature to be overridden on the kernel command line, which may be necessary to work around silicon errata, or to deal with mismatched features on heterogeneous SoC designs, test for CPU feature overrides first, and only then enable LPA2. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 8 ++- arch/arm64/include/asm/cpufeature.h | 18 +++++ arch/arm64/include/asm/memory.h | 4 ++ arch/arm64/kernel/head.S | 8 +++ arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/map_kernel.c | 70 +++++++++++++++++++- arch/arm64/kernel/pi/map_range.c | 11 ++- arch/arm64/kernel/pi/pi.h | 4 +- arch/arm64/mm/init.c | 2 +- arch/arm64/mm/mmu.c | 6 +- arch/arm64/mm/proc.S | 3 + 11 files changed, 124 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 55e8731844cf..d5e139ce0820 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -581,11 +581,17 @@ alternative_endif * but we have to add an offset so that the TTBR1 address corresponds with the * pgdir entry that covers the lowest 48-bit addressable VA. * + * Note that this trick is only used for LVA/64k pages - LPA2/4k pages uses an + * additional paging level, and on LPA2/16k pages, we would end up with a root + * level table with only 2 entries, which is suboptimal in terms of TLB + * utilization, so there we fall back to 47 bits of translation if LPA2 is not + * supported. + * * orr is used as it can cover the immediate value (and is idempotent). * ttbr: Value of ttbr to set, modified. */ .macro offset_ttbr1, ttbr, tmp -#ifdef CONFIG_ARM64_VA_BITS_52 +#if defined(CONFIG_ARM64_VA_BITS_52) && !defined(CONFIG_ARM64_LPA2) mrs \tmp, tcr_el1 and \tmp, \tmp, #TCR_T1SZ_MASK cmp \tmp, #TCR_T1SZ(VA_BITS_MIN) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 351e0b39f0bb..5eed4a12c625 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -990,6 +990,24 @@ static inline bool cpu_has_lva(void) ID_AA64MMFR2_EL1_VARange_SHIFT); } +static inline bool cpu_has_lpa2(void) +{ +#ifdef CONFIG_ARM64_LPA2 + u64 mmfr0; + int feat; + + mmfr0 = read_sysreg(id_aa64mmfr0_el1); + mmfr0 &= ~id_aa64mmfr0_override.mask; + mmfr0 |= id_aa64mmfr0_override.val; + feat = cpuid_feature_extract_signed_field(mmfr0, + ID_AA64MMFR0_EL1_TGRAN_SHIFT); + + return feat >= ID_AA64MMFR0_EL1_TGRAN_LPA2; +#else + return false; +#endif +} + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index f6277f66388f..17ec3068c5f7 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -54,7 +54,11 @@ #define FIXADDR_TOP (-UL(SZ_8M)) #if VA_BITS > 48 +#ifdef CONFIG_ARM64_16K_PAGES +#define VA_BITS_MIN (47) +#else #define VA_BITS_MIN (48) +#endif #else #define VA_BITS_MIN (VA_BITS) #endif diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index e25351addfd0..405e9bce8c73 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -89,6 +89,7 @@ SYM_CODE_START(primary_entry) mov sp, x1 mov x29, xzr adrp x0, init_idmap_pg_dir + mov x1, xzr bl __pi_create_init_idmap /* @@ -473,9 +474,16 @@ SYM_FUNC_END(__enable_mmu) #ifdef CONFIG_ARM64_VA_BITS_52 SYM_FUNC_START(__cpu_secondary_check52bitva) +#ifndef CONFIG_ARM64_LPA2 mrs_s x0, SYS_ID_AA64MMFR2_EL1 and x0, x0, ID_AA64MMFR2_EL1_VARange_MASK cbnz x0, 2f +#else + mrs x0, id_aa64mmfr0_el1 + sbfx x0, x0, #ID_AA64MMFR0_EL1_TGRAN_SHIFT, 4 + cmp x0, #ID_AA64MMFR0_EL1_TGRAN_LPA2 + b.ge 2f +#endif update_early_cpu_boot_status \ CPU_STUCK_IN_KERNEL | CPU_STUCK_REASON_52_BIT_VA, x0, x1 diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index c5eeb33a6280..9472fc3d1303 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -52,6 +52,7 @@ PROVIDE(__pi__ctype = _ctype); PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); PROVIDE(__pi_init_idmap_pg_dir = init_idmap_pg_dir); +PROVIDE(__pi_init_idmap_pg_end = init_idmap_pg_end); PROVIDE(__pi_init_pg_dir = init_pg_dir); PROVIDE(__pi_init_pg_end = init_pg_end); PROVIDE(__pi_swapper_pg_dir = swapper_pg_dir); diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index 18a8d4249136..591394befa5f 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -128,11 +128,64 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) } /* Copy the root page table to its final location */ - memcpy((void *)swapper_pg_dir + va_offset, init_pg_dir, PGD_SIZE); + memcpy((void *)swapper_pg_dir + va_offset, init_pg_dir, PAGE_SIZE); dsb(ishst); idmap_cpu_replace_ttbr1(swapper_pg_dir); } +static void noinline __section(".idmap.text") set_ttbr0_for_lpa2(u64 ttbr) +{ + u64 sctlr = read_sysreg(sctlr_el1); + u64 tcr = read_sysreg(tcr_el1) | TCR_DS; + + asm(" msr sctlr_el1, %0 ;" + " isb ;" + " msr ttbr0_el1, %1 ;" + " msr tcr_el1, %2 ;" + " isb ;" + " tlbi vmalle1 ;" + " dsb nsh ;" + " isb ;" + " msr sctlr_el1, %3 ;" + " isb ;" + :: "r"(sctlr & ~SCTLR_ELx_M), "r"(ttbr), "r"(tcr), "r"(sctlr)); +} + +static void __init remap_idmap_for_lpa2(void) +{ + /* clear the bits that change meaning once LPA2 is turned on */ + pteval_t mask = PTE_SHARED; + + /* + * We have to clear bits [9:8] in all block or page descriptors in the + * initial ID map, as otherwise they will be (mis)interpreted as + * physical address bits once we flick the LPA2 switch (TCR.DS). Since + * we cannot manipulate live descriptors in that way without creating + * potential TLB conflicts, let's create another temporary ID map in a + * LPA2 compatible fashion, and update the initial ID map while running + * from that. + */ + create_init_idmap(init_pg_dir, mask); + dsb(ishst); + set_ttbr0_for_lpa2((u64)init_pg_dir); + + /* + * Recreate the initial ID map with the same granularity as before. + * Don't bother with the FDT, we no longer need it after this. + */ + memset(init_idmap_pg_dir, 0, + (u64)init_idmap_pg_dir - (u64)init_idmap_pg_end); + + create_init_idmap(init_idmap_pg_dir, mask); + dsb(ishst); + + /* switch back to the updated initial ID map */ + set_ttbr0_for_lpa2((u64)init_idmap_pg_dir); + + /* wipe the temporary ID map from memory */ + memset(init_pg_dir, 0, (u64)init_pg_end - (u64)init_pg_dir); +} + static void map_fdt(u64 fdt) { static u8 ptes[INIT_IDMAP_FDT_SIZE] __initdata __aligned(PAGE_SIZE); @@ -155,6 +208,7 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) u64 va_base, pa_base = (u64)&_text; u64 kaslr_offset = pa_base % MIN_KIMG_ALIGN; int root_level = 4 - CONFIG_PGTABLE_LEVELS; + int va_bits = VA_BITS; int chosen; map_fdt((u64)fdt); @@ -166,8 +220,15 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) chosen = fdt_path_offset(fdt, chosen_str); init_feature_override(boot_status, fdt, chosen); - if (VA_BITS > VA_BITS_MIN && cpu_has_lva()) - sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(VA_BITS)); + if (IS_ENABLED(CONFIG_ARM64_64K_PAGES) && !cpu_has_lva()) { + va_bits = VA_BITS_MIN; + } else if (IS_ENABLED(CONFIG_ARM64_LPA2) && !cpu_has_lpa2()) { + va_bits = VA_BITS_MIN; + root_level++; + } + + if (va_bits > VA_BITS_MIN) + sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(va_bits)); /* * The virtual KASLR displacement modulo 2MiB is decided by the @@ -185,6 +246,9 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) kaslr_offset |= kaslr_seed & ~(MIN_KIMG_ALIGN - 1); } + if (IS_ENABLED(CONFIG_ARM64_LPA2) && va_bits > VA_BITS_MIN) + remap_idmap_for_lpa2(); + va_base = KIMAGE_VADDR + kaslr_offset; map_kernel(kaslr_offset, va_base - pa_base, root_level); } diff --git a/arch/arm64/kernel/pi/map_range.c b/arch/arm64/kernel/pi/map_range.c index 79e4f6a2efe1..5410b2cac590 100644 --- a/arch/arm64/kernel/pi/map_range.c +++ b/arch/arm64/kernel/pi/map_range.c @@ -87,14 +87,19 @@ void __init map_range(u64 *pte, u64 start, u64 end, u64 pa, pgprot_t prot, } } -asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir) +asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir, pteval_t clrmask) { u64 ptep = (u64)pg_dir + PAGE_SIZE; + pgprot_t text_prot = PAGE_KERNEL_ROX; + pgprot_t data_prot = PAGE_KERNEL; + + pgprot_val(text_prot) &= ~clrmask; + pgprot_val(data_prot) &= ~clrmask; map_range(&ptep, (u64)_stext, (u64)__initdata_begin, (u64)_stext, - PAGE_KERNEL_ROX, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); + text_prot, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); map_range(&ptep, (u64)__initdata_begin, (u64)_end, (u64)__initdata_begin, - PAGE_KERNEL, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); + data_prot, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); return ptep; } diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h index a8fe79c9b111..93d7457dd10d 100644 --- a/arch/arm64/kernel/pi/pi.h +++ b/arch/arm64/kernel/pi/pi.h @@ -17,7 +17,7 @@ static inline void *prel64_to_pointer(const prel64_t *offset) extern bool dynamic_scs_is_enabled; -extern pgd_t init_idmap_pg_dir[]; +extern pgd_t init_idmap_pg_dir[], init_idmap_pg_end[]; void init_feature_override(u64 boot_status, const void *fdt, int chosen); u64 kaslr_early_init(void *fdt, int chosen); @@ -27,4 +27,4 @@ int scs_patch(const u8 eh_frame[], int size); void map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, int level, pte_t *tbl, bool may_use_cont, u64 va_offset); -asmlinkage u64 create_init_idmap(pgd_t *pgd); +asmlinkage u64 create_init_idmap(pgd_t *pgd, pteval_t clrmask); diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 8a0f8604348b..11c577358d3a 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -361,7 +361,7 @@ void __init arm64_memblock_init(void) * physical address of PAGE_OFFSET, we have to *subtract* from it. */ if (IS_ENABLED(CONFIG_ARM64_VA_BITS_52) && (vabits_actual != 52)) - memstart_addr -= _PAGE_OFFSET(48) - _PAGE_OFFSET(52); + memstart_addr -= _PAGE_OFFSET(vabits_actual) - _PAGE_OFFSET(52); /* * Apply the memory limit if it was set. Since the kernel may be loaded diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 2212289c9a02..ed18cc4cea0d 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -582,8 +582,12 @@ static void __init map_mem(pgd_t *pgdp) * entries at any level are being shared between the linear region and * the vmalloc region. Check whether this is true for the PGD level, in * which case it is guaranteed to be true for all other levels as well. + * (Unless we are running with support for LPA2, in which case the + * entire reduced VA space is covered by a single pgd_t which will have + * been populated without the PXNTable attribute by the time we get here.) */ - BUILD_BUG_ON(pgd_index(direct_map_end - 1) == pgd_index(direct_map_end)); + BUILD_BUG_ON(pgd_index(direct_map_end - 1) == pgd_index(direct_map_end) && + pgd_index(_PAGE_OFFSET(VA_BITS_MIN)) != PTRS_PER_PGD - 1); early_kfence_pool = arm64_kfence_alloc_pool(); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index d60366144031..f32e4b087ef8 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -489,6 +489,9 @@ SYM_FUNC_START(__cpu_setup) mov x9, #64 - VA_BITS alternative_if ARM64_HAS_VA52 tcr_set_t1sz tcr, x9 +#ifdef CONFIG_ARM64_LPA2 + orr tcr, tcr, #TCR_DS +#endif alternative_else_nop_endif #endif From patchwork Tue Sep 12 14:16:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F1A51CA0EEB for ; Tue, 12 Sep 2023 14:21:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=NBVnHPQMoYjpAcz3VJicjAb/fnXrzmgP9Xb6iSfdz2Q=; b=dZWWhFlHm2n7qmwz3Y+BjFI8xq XKSTuCQCxpF4jpJjnqLJ3tp2bRNHdl8gsh+VGk7/0QKzlOvebSarUojVcip0zXPVPj0vIvR4W+VjO QsGy4CtUcJ/0dKxoE7sJPk+BUzmOTRJlN32ihOwa7hF/Raa6wNhrCm2lzkdwrMIvoJ7YJLaGj31YG tbHUtF4sbANGM0pYZVEGDgfdgGbdyV/YYoN8xIa1EdAEl6Q8WI2Kd7Se6s4JGTq3uEEBT38OOMXkJ i1Cz4bvyE2nL/TxduxyRSv30W60Pksm/XPA7LCofbVYHL8pBG6PoZRp5z/yG9Nt0s/mF1vIyfJagu 7funVuLw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Fr-003YSw-3A; Tue, 12 Sep 2023 14:20:40 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Eh-003XTq-0P for linux-arm-kernel@bombadil.infradead.org; Tue, 12 Sep 2023 14:19:27 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=YsvtqLSxpf/VGMm023u50LetdVrmzikeTeAwppCYHyY=; b=PNfVMwyzH9EO5fvgRZ9jAV+bID 3sK3Hvi5TNWnuR5MHYckslsaEXFzF4L91Swk1ODnw3Bhle8UTIgoQsSk9XeBJG8cVHShK0zqtjyBU MrM3tt356ZqXPqESn8E2roh7smXp6BryVTcTckNA7/LEOR4bznJjgMGnu4w9ThAez2aA1Uwq0MkYF C9a86mnY40kansBegbq6FJjYcO8vcC7t4xuU0ePoK0IL7f2o72WtNnEROB+ke59LkoqBhEq8YpkDb 6Dw4yGLMQ2FRQ2bqc+8JVPPBCxmXJqaCRq0OMDv6TrToilY1ToQJ7lnwGnx8G5X8kfVK0roIRGieg 97rh0qgw==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Ea-006A3s-0h for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:25 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-d7fd4c23315so4915329276.2 for ; Tue, 12 Sep 2023 07:19:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528359; x=1695133159; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YsvtqLSxpf/VGMm023u50LetdVrmzikeTeAwppCYHyY=; b=4c+A+9Uc8mIHOwL16NyOvtdxbl++gzTy9NqD09/eOXnpBa6HJBBdNF0wyZfYYHvsJR y1rUzcBik/o6WX847+MiShsyUx8ST9LcV2go/dapv7yHQSSIGTkZ0DDTGDnxVCZMKrJ7 7axmls0b34xi9oIGS2cMdpUXku7kE+CVW5jiJGs1DB/FLFkKQbzjKIiFjgVI6tLWovxU XVGhQP1UuWqq0stC1nCJEcvYbe/CPXFiF8tFcAsTNOQA/P9PaYTmTgzmg6/Lo2Rqdusk LVvW/a3XnORogtlN40+6ZOSD1tIUDZNaXx9mawVgG03AfF8ygm3+QYcTtj9yacfdkwlI gCLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528359; x=1695133159; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YsvtqLSxpf/VGMm023u50LetdVrmzikeTeAwppCYHyY=; b=eCD453w6e2mnVkbC5BdCbkUYlfiXa6Lx/k02EK5SJsf5a6xrEQaqaWI6PSE48uoFtc 2K5VPv4nB+Z6K+FMxR3LWe7HqL82xHDkATVqO80Hwu3zsO8BMIcjNBvosYOzX0VuppA/ j44YtVAaac1b7sHexAbznoFpEf2Q6Wg7+pK2fLa1Y32Bxi2Q4gI/UrPEKwXBBF5W/v+7 RKcqoq6+5CgktSJ620fOptYA5zyrrntHJqWPzHBXXK85sXAkJNyNmj5UWIjsH6H4fv1m rQNyOQVGgOXjbnrS4Xq1vpgWoX0JVf2v5Zsw63Dw5kxlCTVUP93dvw3I8kCDdpV3MY/O ideg== X-Gm-Message-State: AOJu0YwqRMBtXeBtFMBicmFEcqFcPdc0vmqOo5iW0Z3gzIkbyIeC16/2 aoIWwtLqmjTRtyULSZarFOKGIKdk2pmyv3zgqzQqw4LvKUP00IxU7hE7gxomkzTWZErVFGU7pAl pELCHDpMK0CUh/H0bsER95nQqnjQKw5/clYnX8ttVlKqcFAagFuG249L24ZzJxIRvBM3yBThMlJ o= X-Google-Smtp-Source: AGHT+IFuow+IpdEvR+JAf2lutI/rsBOhYW9NAZxTepOqyXLQj+ku/HuAnEZD4uAQFk3wNUvi6arU0c9n X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:68e:b0:d7e:add7:4de6 with SMTP id i14-20020a056902068e00b00d7eadd74de6mr302400ybt.4.1694528359388; Tue, 12 Sep 2023 07:19:19 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:41 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5970; i=ardb@kernel.org; h=from:subject; bh=qLqflDF20wTMiluuLp5NyqSS00r12/n3JMMEtwTs5Q0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6IyYY3N+7aOq+sknPY7fyVz75LdhY2pP1eVVeWx8f mfdHZo6SlkYxDgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwETqNjAyrDnh6PWUvWZHsL57 16uAGI2MrbbGMqdmxDnopLFcUJ9XzMjwzHr5BO8lXxcVME9uCGNMdjZsqdq1zzbgYtHJmwa72X0 ZAA== X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-114-ardb@google.com> Subject: [PATCH v4 51/61] arm64: mm: Add 5 level paging support to fixmap and swapper handling From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_151921_840654_25C72293 X-CRM114-Status: GOOD ( 15.87 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add support for using 5 levels of paging in the fixmap, as well as in the kernel page table handling code which uses fixmaps internally. This also handles the case where a 5 level build runs on hardware that only supports 4 levels of paging. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/fixmap.h | 1 + arch/arm64/include/asm/pgtable.h | 45 ++++++++++++++++--- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/mmu.c | 47 ++++++++++++++++++-- 4 files changed, 85 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/fixmap.h b/arch/arm64/include/asm/fixmap.h index 8aabd45e9a13..87e307804b99 100644 --- a/arch/arm64/include/asm/fixmap.h +++ b/arch/arm64/include/asm/fixmap.h @@ -87,6 +87,7 @@ enum fixed_addresses { FIX_PTE, FIX_PMD, FIX_PUD, + FIX_P4D, FIX_PGD, __end_of_fixed_addresses diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 769abda8a4f9..ad6d129d2098 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -613,12 +613,12 @@ static inline bool pud_table(pud_t pud) { return true; } PUD_TYPE_TABLE) #endif -extern pgd_t init_pg_dir[PTRS_PER_PGD]; +extern pgd_t init_pg_dir[]; extern pgd_t init_pg_end[]; -extern pgd_t swapper_pg_dir[PTRS_PER_PGD]; -extern pgd_t idmap_pg_dir[PTRS_PER_PGD]; -extern pgd_t tramp_pg_dir[PTRS_PER_PGD]; -extern pgd_t reserved_pg_dir[PTRS_PER_PGD]; +extern pgd_t swapper_pg_dir[]; +extern pgd_t idmap_pg_dir[]; +extern pgd_t tramp_pg_dir[]; +extern pgd_t reserved_pg_dir[]; extern void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd); @@ -883,12 +883,47 @@ static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long addr) return p4d_offset_lockless(pgdp, READ_ONCE(*pgdp), addr); } +static inline p4d_t *p4d_set_fixmap(unsigned long addr) +{ + if (!pgtable_l5_enabled()) + return NULL; + return (p4d_t *)set_fixmap_offset(FIX_P4D, addr); +} + +static inline p4d_t *p4d_set_fixmap_offset(pgd_t *pgdp, unsigned long addr) +{ + if (!pgtable_l5_enabled()) + return pgd_to_folded_p4d(pgdp, addr); + return p4d_set_fixmap(p4d_offset_phys(pgdp, addr)); +} + +static inline void p4d_clear_fixmap(void) +{ + if (pgtable_l5_enabled()) + clear_fixmap(FIX_P4D); +} + +/* use ONLY for statically allocated translation tables */ +static inline p4d_t *p4d_offset_kimg(pgd_t *pgdp, u64 addr) +{ + if (!pgtable_l5_enabled()) + return pgd_to_folded_p4d(pgdp, addr); + return (p4d_t *)__phys_to_kimg(p4d_offset_phys(pgdp, addr)); +} + #define pgd_page(pgd) pfn_to_page(__phys_to_pfn(__pgd_to_phys(pgd))) #else static inline bool pgtable_l5_enabled(void) { return false; } +/* Match p4d_offset folding in */ +#define p4d_set_fixmap(addr) NULL +#define p4d_set_fixmap_offset(p4dp, addr) ((p4d_t *)p4dp) +#define p4d_clear_fixmap() + +#define p4d_offset_kimg(dir,addr) ((p4d_t *)dir) + #endif /* CONFIG_PGTABLE_LEVELS > 4 */ #define pgd_ERROR(e) \ diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index 9436a12e1882..355ad694c5ef 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -101,7 +101,7 @@ void __init early_fixmap_init(void) unsigned long end = FIXADDR_TOP; pgd_t *pgdp = pgd_offset_k(addr); - p4d_t *p4dp = p4d_offset(pgdp, addr); + p4d_t *p4dp = p4d_offset_kimg(pgdp, addr); early_fixmap_init_pud(p4dp, addr, end); } diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index ed18cc4cea0d..916f4b1814ff 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -313,15 +313,14 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, } while (addr = next, addr != end); } -static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, +static void alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; - pud_t *pudp; - p4d_t *p4dp = p4d_offset(pgdp, addr); p4d_t p4d = READ_ONCE(*p4dp); + pud_t *pudp; if (p4d_none(p4d)) { p4dval_t p4dval = P4D_TYPE_TABLE | P4D_TABLE_UXN; @@ -369,6 +368,46 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, pud_clear_fixmap(); } +static void alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + unsigned long next; + pgd_t pgd = READ_ONCE(*pgdp); + p4d_t *p4dp; + + if (pgd_none(pgd)) { + pgdval_t pgdval = PGD_TYPE_TABLE | PGD_TABLE_UXN; + phys_addr_t p4d_phys; + + if (flags & NO_EXEC_MAPPINGS) + pgdval |= PGD_TABLE_PXN; + BUG_ON(!pgtable_alloc); + p4d_phys = pgtable_alloc(P4D_SHIFT); + __pgd_populate(pgdp, p4d_phys, pgdval); + pgd = READ_ONCE(*pgdp); + } + BUG_ON(pgd_bad(pgd)); + + p4dp = p4d_set_fixmap_offset(pgdp, addr); + do { + p4d_t old_p4d = READ_ONCE(*p4dp); + + next = p4d_addr_end(addr, end); + + alloc_init_pud(p4dp, addr, next, phys, prot, + pgtable_alloc, flags); + + BUG_ON(p4d_val(old_p4d) != 0 && + p4d_val(old_p4d) != READ_ONCE(p4d_val(*p4dp))); + + phys += next - addr; + } while (p4dp++, addr = next, addr != end); + + p4d_clear_fixmap(); +} + static void __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, @@ -391,7 +430,7 @@ static void __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, do { next = pgd_addr_end(addr, end); - alloc_init_pud(pgdp, addr, next, phys, prot, pgtable_alloc, + alloc_init_p4d(pgdp, addr, next, phys, prot, pgtable_alloc, flags); phys += next - addr; } while (pgdp++, addr = next, addr != end); From patchwork Tue Sep 12 14:16:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381810 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 99E5DCA0EEC for ; Tue, 12 Sep 2023 14:21:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=qVrOJsAqTqDQFGlKvfjTOKYvcyVC1wLlkVg4AlMr7xI=; b=nkv09k565Sm/CPrqj/Nx9RrpxZ sc9A3Q5aB0048XRQvQLSouH1fD9ToX24chvCkuoX1e+abjC36pn9soxEUo0UD8GZEBze08IaHQEMX DrYgQNxupVmiBMWLuyCUwwv3eBclTq+2Jqgw/buWQon2RBtTJuumEfqY9dfAvHYIZh9Vm7lYhMBb8 swBQQXtvDQXGmIHu54m9XBdqwbIasYsGE8B8Wj9DbKDyquaC3dHNf/kIvajIheoferNcSNAptCesa phCNnYRJeCV6Hug6rYPMAzZX0WrYpskxtRSoQDQle6qoF0LnhhICgJgOSveTc9qAH99uge+KVI53w wx1u4Egw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Fw-003YXL-22; Tue, 12 Sep 2023 14:20:44 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Ed-003XPp-1r for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:31 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-31f46ccee0fso2672610f8f.1 for ; Tue, 12 Sep 2023 07:19:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528361; x=1695133161; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=c8zKIpc5iFIdtUUDKjAsnLhzWnZMeeOT58mbydS7ulU=; b=I5GHGB4Q0BFhOAfb5kUu/bGxgMsbdo074sZI7T+05FeY+tAZjmKDhhwcvIxTU4qUwq Gi2VngtjHEXCqHoUJBFz10GYAjD59cDiTF+ZkRn8vjAnNka3z/CN5gDcNa4V+ygSXmZ5 UGOVoL+dp2194SPJsga1y+xVMh0qDsQ7S7LldX7HKW+60Kj+zgKgm9SRkiWQHTe08XPB VLUVsAqIjDd+M1PQBWd7nxSocohmndR5LdN8RdRZ9dBUg8vtEs8gaiaCG4trBJsukOiS wsAf9AkSFfq1ZtcxV5NHyrOE/n41wSXTdd21caC8C6bw5RJGNinDEL+WNpfpoof1jrfk Auew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528361; x=1695133161; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=c8zKIpc5iFIdtUUDKjAsnLhzWnZMeeOT58mbydS7ulU=; b=FasrddP6f9fKhl83vUna5gSC9lKwsgkoPzdNToY8d3ugr5cE12gq+hEtpnhV9WxOwX RqhbdlUYm00ZiLuxYewl3C4m1866bEpESzAauVRzwEPr1FsqeuXpMlsldEPNWgQMWJBw E9/CC9MAe/PkgtPctGfBKaw/P/y7bA7g0zkxztdgzxwiIe7V8idREqRWDlonpqqKkO88 uqzmTDeUeP7BLDq/Lmwu+AI+Z6doVrq0pvU5Or6cbWHO+p67dRotU+v6cmz/HHv+F/YV dnHQ1j5lHT5f6wgdxyPpuC//C4PAfSsp4humJknxpjg3Rd5FukfF7EEUoQoEF4nHqf9B n3IA== X-Gm-Message-State: AOJu0YxqJPxbojyqL/XVvOUDgDOT0Vcy5Ue9rNLbjYhKT/TaiZeC9mdh Bl15HddJ7EL0JP1I/3f1rypKEa1DGIsJSYJ0hdkI7Ausz6lS4ZGmXvFPF+axe6gTjxUrT7oMu1x 8o6FX/KcPztr5aGfHqd/S8VXS7hPwpeoag4CAOir1ZtYiZLfuvXGAq83L+XGwGsMmsUYm1kMlWR M= X-Google-Smtp-Source: AGHT+IF4I31TK7tVp4s9amxZouYsuh3lHD+ECcmU/Y70rcZ+lWDT7RE+69I6rWFVq/E8pJzyMfX3LbDO X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:e741:0:b0:31a:d13b:bfcd with SMTP id c1-20020adfe741000000b0031ad13bbfcdmr47503wrn.7.1694528361558; Tue, 12 Sep 2023 07:19:21 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:42 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=9254; i=ardb@kernel.org; h=from:subject; bh=Y2d+uUuL2e1k+6iT7TOaRtQFSZKWETgYT5u9Cbsi9zo=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6Kz+dze2LTsXx189Janocvlis5Tw7u97sxiSnr2Py 1uzLiC1o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAExE7zrDP73V+Syn3h5peL5b P09uc00a9y//uYvKGAQtbrSZNVzlYWZk+B48Zd2XGWtF5rsEvJinf9bncfHJQDEW59ZPfSacKrc 9WAA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-115-ardb@google.com> Subject: [PATCH v4 52/61] arm64: kasan: Reduce minimum shadow alignment and enable 5 level paging From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071923_691457_1AB7BB66 X-CRM114-Status: GOOD ( 25.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Allow the KASAN init code to deal with 5 levels of paging, and relax the requirement that the shadow region is aligned to the top level pgd_t size. This is necessary for LPA2 based 52-bit virtual addressing, where the KASAN shadow will never be aligned to the pgd_t size. Allowing this also enables the 16k/48-bit case for KASAN, which is a nice bonus. This involves some hackery to manipulate the root and next level page tables without having to distinguish all the various configurations, including 16k/48-bits (which has a two entry pgd_t level), and LPA2 configurations running with one translation level less on non-LPA2 hardware. Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 2 +- arch/arm64/mm/kasan_init.c | 143 ++++++++++++++++++-- 2 files changed, 130 insertions(+), 15 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a3f062605fa9..3a8d40a2adb7 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -162,7 +162,7 @@ config ARM64 select HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_JUMP_LABEL_RELATIVE - select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) + select HAVE_ARCH_KASAN select HAVE_ARCH_KASAN_VMALLOC if HAVE_ARCH_KASAN select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN select HAVE_ARCH_KASAN_HW_TAGS if (HAVE_ARCH_KASAN && ARM64_MTE) diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 82798dcd6323..a1a939507a68 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -23,7 +23,7 @@ #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) -static pgd_t tmp_pg_dir[PTRS_PER_PGD] __initdata __aligned(PGD_SIZE); +static pgd_t tmp_pg_dir[PTRS_PER_PTE] __initdata __aligned(PAGE_SIZE); /* * The p*d_populate functions call virt_to_phys implicitly so they can't be used @@ -99,6 +99,19 @@ static pud_t *__init kasan_pud_offset(p4d_t *p4dp, unsigned long addr, int node, return early ? pud_offset_kimg(p4dp, addr) : pud_offset(p4dp, addr); } +static p4d_t *__init kasan_p4d_offset(pgd_t *pgdp, unsigned long addr, int node, + bool early) +{ + if (pgd_none(READ_ONCE(*pgdp))) { + phys_addr_t p4d_phys = early ? + __pa_symbol(kasan_early_shadow_p4d) + : kasan_alloc_zeroed_page(node); + __pgd_populate(pgdp, p4d_phys, PGD_TYPE_TABLE); + } + + return early ? p4d_offset_kimg(pgdp, addr) : p4d_offset(pgdp, addr); +} + static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, unsigned long end, int node, bool early) { @@ -144,12 +157,12 @@ static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr, unsigned long end, int node, bool early) { unsigned long next; - p4d_t *p4dp = p4d_offset(pgdp, addr); + p4d_t *p4dp = kasan_p4d_offset(pgdp, addr, node, early); do { next = p4d_addr_end(addr, end); kasan_pud_populate(p4dp, addr, next, node, early); - } while (p4dp++, addr = next, addr != end); + } while (p4dp++, addr = next, addr != end && p4d_none(READ_ONCE(*p4dp))); } static void __init kasan_pgd_populate(unsigned long addr, unsigned long end, @@ -165,14 +178,48 @@ static void __init kasan_pgd_populate(unsigned long addr, unsigned long end, } while (pgdp++, addr = next, addr != end); } +#if defined(CONFIG_ARM64_64K_PAGES) || CONFIG_PGTABLE_LEVELS > 4 +#define SHADOW_ALIGN P4D_SIZE +#else +#define SHADOW_ALIGN PUD_SIZE +#endif + +/* + * Return whether 'addr' is aligned to the size covered by a root level + * descriptor. + */ +static bool __init root_level_aligned(u64 addr) +{ + int shift = (ARM64_HW_PGTABLE_LEVELS(vabits_actual) - 1) * (PAGE_SHIFT - 3); + + return (addr % (PAGE_SIZE << shift)) == 0; +} + /* The early shadow maps everything to a single page of zeroes */ asmlinkage void __init kasan_early_init(void) { BUILD_BUG_ON(KASAN_SHADOW_OFFSET != KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); - BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS), PGDIR_SIZE)); - BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS_MIN), PGDIR_SIZE)); - BUILD_BUG_ON(!IS_ALIGNED(KASAN_SHADOW_END, PGDIR_SIZE)); + BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS), SHADOW_ALIGN)); + BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS_MIN), SHADOW_ALIGN)); + BUILD_BUG_ON(!IS_ALIGNED(KASAN_SHADOW_END, SHADOW_ALIGN)); + + if (!root_level_aligned(KASAN_SHADOW_START)) { + /* + * The start address is misaligned, and so the next level table + * will be shared with the linear region. This can happen with + * 4 or 5 level paging, so install a generic pte_t[] as the + * next level. This prevents the kasan_pgd_populate call below + * from inserting an entry that refers to the shared KASAN zero + * shadow pud_t[]/p4d_t[], which could end up getting corrupted + * when the linear region is mapped. + */ + static pte_t tbl[PTRS_PER_PTE] __page_aligned_bss; + pgd_t *pgdp = pgd_offset_k(KASAN_SHADOW_START); + + set_pgd(pgdp, __pgd(__pa_symbol(tbl) | PGD_TYPE_TABLE)); + } + kasan_pgd_populate(KASAN_SHADOW_START, KASAN_SHADOW_END, NUMA_NO_NODE, true); } @@ -184,20 +231,75 @@ static void __init kasan_map_populate(unsigned long start, unsigned long end, kasan_pgd_populate(start & PAGE_MASK, PAGE_ALIGN(end), node, false); } -static void __init clear_pgds(unsigned long start, - unsigned long end) +/* + * Return the descriptor index of 'addr' in the root level table + */ +static int __init root_level_idx(u64 addr) { /* - * Remove references to kasan page tables from - * swapper_pg_dir. pgd_clear() can't be used - * here because it's nop on 2,3-level pagetable setups + * On 64k pages, the TTBR1 range root tables are extended for 52-bit + * virtual addressing, and TTBR1 will simply point to the pgd_t entry + * that covers the start of the 48-bit addressable VA space if LVA is + * not implemented. This means we need to index the table as usual, + * instead of masking off bits based on vabits_actual. */ - for (; start < end; start += PGDIR_SIZE) - set_pgd(pgd_offset_k(start), __pgd(0)); + u64 vabits = IS_ENABLED(CONFIG_ARM64_64K_PAGES) ? VA_BITS + : vabits_actual; + int shift = (ARM64_HW_PGTABLE_LEVELS(vabits) - 1) * (PAGE_SHIFT - 3); + + return (addr & ~_PAGE_OFFSET(vabits)) >> (shift + PAGE_SHIFT); +} + +/* + * Clone a next level table from swapper_pg_dir into tmp_pg_dir + */ +static void __init clone_next_level(u64 addr, pgd_t *tmp_pg_dir, pud_t *pud) +{ + int idx = root_level_idx(addr); + pgd_t pgd = READ_ONCE(swapper_pg_dir[idx]); + pud_t *pudp = (pud_t *)__phys_to_kimg(__pgd_to_phys(pgd)); + + memcpy(pud, pudp, PAGE_SIZE); + tmp_pg_dir[idx] = __pgd(__phys_to_pgd_val(__pa_symbol(pud)) | + PUD_TYPE_TABLE); +} + +/* + * Return the descriptor index of 'addr' in the next level table + */ +static int __init next_level_idx(u64 addr) +{ + int shift = (ARM64_HW_PGTABLE_LEVELS(vabits_actual) - 2) * (PAGE_SHIFT - 3); + + return (addr >> (shift + PAGE_SHIFT)) % PTRS_PER_PTE; +} + +/* + * Dereference the table descriptor at 'pgd_idx' and clear the entries from + * 'start' to 'end' (exclusive) from the table. + */ +static void __init clear_next_level(int pgd_idx, int start, int end) +{ + pgd_t pgd = READ_ONCE(swapper_pg_dir[pgd_idx]); + pud_t *pudp = (pud_t *)__phys_to_kimg(__pgd_to_phys(pgd)); + + memset(&pudp[start], 0, (end - start) * sizeof(pud_t)); +} + +static void __init clear_shadow(u64 start, u64 end) +{ + int l = root_level_idx(start), m = root_level_idx(end); + + if (!root_level_aligned(start)) + clear_next_level(l++, next_level_idx(start), PTRS_PER_PTE); + if (!root_level_aligned(end)) + clear_next_level(m, 0, next_level_idx(end)); + memset(&swapper_pg_dir[l], 0, (m - l) * sizeof(pgd_t)); } static void __init kasan_init_shadow(void) { + static pud_t pud[2][PTRS_PER_PUD] __initdata __aligned(PAGE_SIZE); u64 kimg_shadow_start, kimg_shadow_end; u64 mod_shadow_start; u64 vmalloc_shadow_end; @@ -219,10 +321,23 @@ static void __init kasan_init_shadow(void) * setup will be finished. */ memcpy(tmp_pg_dir, swapper_pg_dir, sizeof(tmp_pg_dir)); + + /* + * If the start or end address of the shadow region is not aligned to + * the root level size, we have to allocate a temporary next-level table + * in each case, clone the next level of descriptors, and install the + * table into tmp_pg_dir. Note that with 5 levels of paging, the next + * level will in fact be p4d_t, but that makes no difference in this + * case. + */ + if (!root_level_aligned(KASAN_SHADOW_START)) + clone_next_level(KASAN_SHADOW_START, tmp_pg_dir, pud[0]); + if (!root_level_aligned(KASAN_SHADOW_END)) + clone_next_level(KASAN_SHADOW_END, tmp_pg_dir, pud[1]); dsb(ishst); cpu_replace_ttbr1(lm_alias(tmp_pg_dir)); - clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); + clear_shadow(KASAN_SHADOW_START, KASAN_SHADOW_END); kasan_map_populate(kimg_shadow_start, kimg_shadow_end, early_pfn_to_nid(virt_to_pfn(lm_alias(KERNEL_START)))); From patchwork Tue Sep 12 14:16:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2D8EACA0EEC for ; Tue, 12 Sep 2023 14:21:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=XG+TLGxdBsxAdrpvGKAQwTkwN2z0smh4pMchnZoJDpc=; b=g9ceauq4O2fF8DnseF7eMV9di3 si8SAhTU+dM60yQHDozqGI1FrV7sJ8HsEsYrVmyYjA8G5CcuY4CeYJSkay5d694c4p/lDmC4aGYAD 3Xhts5vuTVUK3d8fzwUwRcwfvcJUOFvvkvsqXsoMbE5zB7AYbQwQZjqakayQSxU2XjEI6Wza6gVHu HQyWwBBhRiKJArrFLYbQvBesw0tcMamPBcdCaSv6LRvxaumnlOYTlO6wpLTAzZZoO3ghVL0VgD7Cb t9O7A+BL2nk+9MnN4GgFXdVPS8kTB5nloy1JZrYSUnHSba8U/Qm4O9X4bbxdx+qNgGQSOhWLvg+Fu TJRXVb0Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Fz-003Ya4-0Y; Tue, 12 Sep 2023 14:20:47 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Ef-003XS6-1z for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:32 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-58fc448ee4fso62003507b3.2 for ; Tue, 12 Sep 2023 07:19:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528364; x=1695133164; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xMKu/PDkl1bV6C+qaNExH/s5/xIXY/8eWGg/uduUfzs=; b=DPH63oAOQSV8GvC21hwUteA0w8Ccpsj1QTTMj2lsS/lPSakEPhvFizXx4ZNwmefEE3 ldy8wLl3iYhawC6QQLAUbz73Z+JkkiEgRfr9FgInmSKAh/aTBbvV4ni6FL6+bf7yya9O U+Bi5ExuSlwkGZgoMOeoRr4GZDOqa1GXe/YuJGbpnVSDermrV6X6ckMABqbhFhtH4vnZ VS53xH7wIo0MVxsLHMuMzi1doKAWEwh0I6SgtHyqCoK346LX2PRmv8unFPbftwasQbdI eMOsOXazNMpv1n9OkY1rT0/okB1eFcmeh5HP1OVCFnGcFzFEsDbHwVfO8XdvOtEvIQ44 gwwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528364; x=1695133164; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xMKu/PDkl1bV6C+qaNExH/s5/xIXY/8eWGg/uduUfzs=; b=TcXSPtj6N7tq6/HOiMemG/Dzh6NZjXQ9qbRyYiVNvBNKue6Oysk9zn2Nx9jvah8h2W OHsDcETlGWe1MKJDkhpSUPa6/qA2kwkWCDil9jsOn3sNtyWvVF+xewuxYvBvs8CTBcgd RAetIlx+Priy6NOLNFR6iUzYHh0SAVX9iGZPilhuiPVv18k6JZqNqJBNS93K37PV1Fa9 RAymHR82mDTI/9BxelKZs63H9ZK4qjh6u80lLCsocHE4HbDTFVgLNm1LVphnv+Bhn1dH Nl9PxneRglJCNAcroeRSOf+aveYYHNMdAipCA8YGq6zRMtcji145+fU8cIDb0+s0axnt 1f0A== X-Gm-Message-State: AOJu0YzPTDhBtUYrgAYQihbEwNV+nR/ZAS/Q/rZD9nq11EfOev/Ktk5N 3fwRfmSmAr4yT3kHcOVNuk0aczUB/vyG44aHYqKro3YeopASgXUJ5r7lOWv/HF7tD1vUjgYMps4 F/i1XxEsy7RWzqcC+/w8N9fnlVhQtlTvfX0i3JkUpmwzo9xhODzZMz3HzB6jCxPW52DWBNX+a3u w= X-Google-Smtp-Source: AGHT+IGgyrGxxHOMRHkYHLuXkvDEIhZ9HrwYesQPybvmlek4QixAmjJGCUeAuLPArbQHhTrwWJ8w+HWg X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:161a:b0:d74:93a1:70a2 with SMTP id bw26-20020a056902161a00b00d7493a170a2mr303747ybb.5.1694528364116; Tue, 12 Sep 2023 07:19:24 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:43 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=8345; i=ardb@kernel.org; h=from:subject; bh=VmyrKh1STAt+BFEhXsK2eT2JjP45zAPMlP8+07Gapn0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6FzA8xc107Tc909+PNk3pPo5207nWNs/OfwCXS8Of lOYYVHXUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACbSfpORYU6xfOe1rsXzDU4n VUn0774VY1DUY6r+5vzsOeHcqrM45zEyrFsjzCca/WeS69W00+liiQnq0SzW/5Mrp+S952S8Y7S bCwA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-116-ardb@google.com> Subject: [PATCH v4 53/61] arm64: mm: Add support for folding PUDs at runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071925_681673_51E97431 X-CRM114-Status: GOOD ( 19.77 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In order to support LPA2 on 16k pages in a way that permits non-LPA2 systems to run the same kernel image, we have to be able to fall back to at most 48 bits of virtual addressing. Falling back to 48 bits would result in a level 0 with only 2 entries, which is suboptimal in terms of TLB utilization. So instead, let's fall back to 47 bits in that case. This means we need to be able to fold PUDs dynamically, similar to how we fold P4Ds for 48 bit virtual addressing on LPA2 with 4k pages. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/pgalloc.h | 12 ++- arch/arm64/include/asm/pgtable.h | 87 +++++++++++++++++--- arch/arm64/include/asm/tlb.h | 3 +- arch/arm64/kernel/cpufeature.c | 2 + arch/arm64/mm/mmu.c | 2 +- arch/arm64/mm/pgd.c | 2 + 6 files changed, 94 insertions(+), 14 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index cae8c648f462..aeba2cf15a25 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -14,6 +14,7 @@ #include #define __HAVE_ARCH_PGD_FREE +#define __HAVE_ARCH_PUD_FREE #include #define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) @@ -43,7 +44,8 @@ static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t prot) static inline void __p4d_populate(p4d_t *p4dp, phys_addr_t pudp, p4dval_t prot) { - set_p4d(p4dp, __p4d(__phys_to_p4d_val(pudp) | prot)); + if (pgtable_l4_enabled()) + set_p4d(p4dp, __p4d(__phys_to_p4d_val(pudp) | prot)); } static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4dp, pud_t *pudp) @@ -53,6 +55,14 @@ static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4dp, pud_t *pudp) p4dval |= (mm == &init_mm) ? P4D_TABLE_UXN : P4D_TABLE_PXN; __p4d_populate(p4dp, __pa(pudp), p4dval); } + +static inline void pud_free(struct mm_struct *mm, pud_t *pud) +{ + if (!pgtable_l4_enabled()) + return; + BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); + free_page((unsigned long)pud); +} #else static inline void __p4d_populate(p4d_t *p4dp, phys_addr_t pudp, p4dval_t prot) { diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index ad6d129d2098..a66792d468f9 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -751,12 +751,27 @@ static inline pmd_t *pud_pgtable(pud_t pud) #if CONFIG_PGTABLE_LEVELS > 3 +static __always_inline bool pgtable_l4_enabled(void) +{ + if (CONFIG_PGTABLE_LEVELS > 4 || !IS_ENABLED(CONFIG_ARM64_LPA2)) + return true; + if (!alternative_has_cap_likely(ARM64_ALWAYS_BOOT)) + return vabits_actual == VA_BITS; + return alternative_has_cap_unlikely(ARM64_HAS_VA52); +} + +static inline bool mm_pud_folded(const struct mm_struct *mm) +{ + return !pgtable_l4_enabled(); +} +#define mm_pud_folded mm_pud_folded + #define pud_ERROR(e) \ pr_err("%s:%d: bad pud %016llx.\n", __FILE__, __LINE__, pud_val(e)) -#define p4d_none(p4d) (!p4d_val(p4d)) -#define p4d_bad(p4d) (!(p4d_val(p4d) & 2)) -#define p4d_present(p4d) (p4d_val(p4d)) +#define p4d_none(p4d) (pgtable_l4_enabled() && !p4d_val(p4d)) +#define p4d_bad(p4d) (pgtable_l4_enabled() && !(p4d_val(p4d) & 2)) +#define p4d_present(p4d) (!p4d_none(p4d)) static inline void set_p4d(p4d_t *p4dp, p4d_t p4d) { @@ -772,7 +787,8 @@ static inline void set_p4d(p4d_t *p4dp, p4d_t p4d) static inline void p4d_clear(p4d_t *p4dp) { - set_p4d(p4dp, __p4d(0)); + if (pgtable_l4_enabled()) + set_p4d(p4dp, __p4d(0)); } static inline phys_addr_t p4d_page_paddr(p4d_t p4d) @@ -780,25 +796,74 @@ static inline phys_addr_t p4d_page_paddr(p4d_t p4d) return __p4d_to_phys(p4d); } +#define pud_index(addr) (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1)) + +static inline pud_t *p4d_to_folded_pud(p4d_t *p4dp, unsigned long addr) +{ + return (pud_t *)PTR_ALIGN_DOWN(p4dp, PAGE_SIZE) + pud_index(addr); +} + static inline pud_t *p4d_pgtable(p4d_t p4d) { return (pud_t *)__va(p4d_page_paddr(p4d)); } -/* Find an entry in the first-level page table. */ -#define pud_offset_phys(dir, addr) (p4d_page_paddr(READ_ONCE(*(dir))) + pud_index(addr) * sizeof(pud_t)) +static inline phys_addr_t pud_offset_phys(p4d_t *p4dp, unsigned long addr) +{ + BUG_ON(!pgtable_l4_enabled()); -#define pud_set_fixmap(addr) ((pud_t *)set_fixmap_offset(FIX_PUD, addr)) -#define pud_set_fixmap_offset(p4d, addr) pud_set_fixmap(pud_offset_phys(p4d, addr)) -#define pud_clear_fixmap() clear_fixmap(FIX_PUD) + return p4d_page_paddr(READ_ONCE(*p4dp)) + pud_index(addr) * sizeof(pud_t); +} -#define p4d_page(p4d) pfn_to_page(__phys_to_pfn(__p4d_to_phys(p4d))) +static inline +pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsigned long addr) +{ + if (!pgtable_l4_enabled()) + return p4d_to_folded_pud(p4dp, addr); + return (pud_t *)__va(p4d_page_paddr(p4d)) + pud_index(addr); +} +#define pud_offset_lockless pud_offset_lockless + +static inline pud_t *pud_offset(p4d_t *p4dp, unsigned long addr) +{ + return pud_offset_lockless(p4dp, READ_ONCE(*p4dp), addr); +} +#define pud_offset pud_offset + +static inline pud_t *pud_set_fixmap(unsigned long addr) +{ + if (!pgtable_l4_enabled()) + return NULL; + return (pud_t *)set_fixmap_offset(FIX_PUD, addr); +} + +static inline pud_t *pud_set_fixmap_offset(p4d_t *p4dp, unsigned long addr) +{ + if (!pgtable_l4_enabled()) + return p4d_to_folded_pud(p4dp, addr); + return pud_set_fixmap(pud_offset_phys(p4dp, addr)); +} + +static inline void pud_clear_fixmap(void) +{ + if (pgtable_l4_enabled()) + clear_fixmap(FIX_PUD); +} /* use ONLY for statically allocated translation tables */ -#define pud_offset_kimg(dir,addr) ((pud_t *)__phys_to_kimg(pud_offset_phys((dir), (addr)))) +static inline pud_t *pud_offset_kimg(p4d_t *p4dp, u64 addr) +{ + if (!pgtable_l4_enabled()) + return p4d_to_folded_pud(p4dp, addr); + return (pud_t *)__phys_to_kimg(pud_offset_phys(p4dp, addr)); +} + +#define p4d_page(p4d) pfn_to_page(__phys_to_pfn(__p4d_to_phys(p4d))) #else +static inline bool pgtable_l4_enabled(void) { return false; } + #define p4d_page_paddr(p4d) ({ BUILD_BUG(); 0;}) /* Match pud_offset folding in */ diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 2c29239d05c3..771077f34a6c 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -96,7 +96,8 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, unsigned long addr) { - tlb_remove_ptdesc(tlb, virt_to_ptdesc(pudp)); + if (pgtable_l4_enabled()) + tlb_remove_ptdesc(tlb, virt_to_ptdesc(pudp)); } #endif diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index f4c81300cb13..778f42cfcf90 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1755,6 +1755,8 @@ kpti_install_ng_mappings(const struct arm64_cpu_capabilities *__unused) if (levels == 5 && !pgtable_l5_enabled()) levels = 4; + else if (levels == 4 && !pgtable_l4_enabled()) + levels = 3; remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 916f4b1814ff..3665a0578ba8 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1062,7 +1062,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr, free_empty_pmd_table(pudp, addr, next, floor, ceiling); } while (addr = next, addr < end); - if (CONFIG_PGTABLE_LEVELS <= 3) + if (!pgtable_l4_enabled()) return; if (!pgtable_range_aligned(start, end, floor, ceiling, P4D_MASK)) diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c index 3c4f8a279d2b..0c501cabc238 100644 --- a/arch/arm64/mm/pgd.c +++ b/arch/arm64/mm/pgd.c @@ -21,6 +21,8 @@ static bool pgdir_is_page_size(void) { if (PGD_SIZE == PAGE_SIZE) return true; + if (CONFIG_PGTABLE_LEVELS == 4) + return !pgtable_l4_enabled(); if (CONFIG_PGTABLE_LEVELS == 5) return !pgtable_l5_enabled(); return false; From patchwork Tue Sep 12 14:16:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381864 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BDCC1CA0EF5 for ; Tue, 12 Sep 2023 15:30:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=UaK+flF+GeMESrwGxjU6hsIJQlqnB8NvcgnPpyvtpGM=; b=nyu4IKLyXEGsNBeQL5z3THAZsH 3i6JbEpHAXJ+q1gOrMGhdzCFOMAPE/33nXCYVdZkBefFl8EOqA5yE7jSpaS6hQWjntXHg+C+ARVxH /E4yMsV1IkViTSD7rDCo+SIbDncCysidfiRbQXt/uii1ue0uFMS+fytUUmMa7aosZ7WbccBzJvloz nbVRLroxMU6BqA+urObAY+jxwFsV0b3YglDpJm765wwYtiFkkGFETtst7c77iRZ2RXqbhb9st8Q5v Hp6v+Zlt1J3RmGj4jI5cX4+E2lwAmSDHJI3Lx7ws2ma+WqcNQm5iFNzjQXJW9NYJavhoEQggH3KpW bwbUPKVg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg5LR-003ihd-1k; Tue, 12 Sep 2023 15:30:29 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Ei-003XUd-21 for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:34 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-31fa1631b0eso1663443f8f.2 for ; Tue, 12 Sep 2023 07:19:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528366; x=1695133166; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2q1VRrjkhHjbG6wFyzvSTGEX790OB5iXPWlDPWUoBkw=; b=y+LVHXhqMtroXhHzaXCMBpPB0X4ANWoEU2EKHojPVw9a5EmhK5Z2UxZn1ZJn/0Q5j2 BD5NJyMv2k47/zJRKrajr+Vf/rwr5kIVx5MMtn+8y3AV9RMKAw907+4rKimQRNNk/70N slVcBe4dQZTTFyY+WyGD8zg2gCXrtNn0xBmoA+0cd0aOU29uQqvSIBxNc6GYosCEmAma +0cmoZuLj1F0v3idylH5inplSnNtjHzn9GkPAUGxpR12BjZ1C/kgMzGlSY37AXBRWCQF vrr6U2u0grjOui0O99RxFKXuC3LYz402qj3Sf2Igbl8m5lewxvSz6JBy9W9157PfnQq5 jefw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528366; x=1695133166; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2q1VRrjkhHjbG6wFyzvSTGEX790OB5iXPWlDPWUoBkw=; b=F40Lvnfr9Rn2OvE8TEYDkjiTKi9/2RHPYzDC+W1Flm9Q5wj6rgh7Y+wXINTkc1/hxP JcJmqM1yXYN0ZESgrqandHyp7pILiJi8xrreDy6rBxrcijz24FvfaSfJ69OZw8YBRoV+ /g8eBXCobEllwWsNTxn9sMJcm+DWUT/7ZbnEcdwpfvT8ol/D5v4P1+6vsmPlrlztd75M GVEELzBuB1bJWwWmprPRwXxRngYDgl4Atvr3ElE0R/idK/qsbvomseVaiB25GUE1v6CI MlW9JQsCRVxhopDoBAkNkVdUFFss3HEq2t9sz42/e2VF/BJTUVwPC295avf6flBdeNk5 v/Wg== X-Gm-Message-State: AOJu0Yyl7/vQg49DTwhTX+iWj/fovxqgWoypR4jAPEjGQqD+7toebFen 3x0DmJILRTRNflwMlc09WkWZb3bv12CKmsYIGFwjfEnUK/SWd86Jf6/DyfMzlS/PQUqAp106Eqp SOVDwdAgFvs/UNLOLWlkEy8LzvfRijJj6L5mQcmYRN4ytAxRQmvrPUUy2b4JifSZeuv3x+NTluW E= X-Google-Smtp-Source: AGHT+IGM8fG/aQOjNu+C/ESUxXvXEUyqDjS+4sjTOyG8sRg3KPt9fi9+kxE3WeGAavzNNNQGsRc4JyTO X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:fc0d:0:b0:318:1210:210b with SMTP id i13-20020adffc0d000000b003181210210bmr183719wrr.12.1694528366412; Tue, 12 Sep 2023 07:19:26 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:44 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1822; i=ardb@kernel.org; h=from:subject; bh=CRvId4fOeua2UCLYyJMWyj3lTxC7N11/FgVB0VS/pms=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6Hzwsarzh4umPxDxSTtr4PgygvevnqfkLf19wXPjK q76ptV1lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgImIzmT4p9O1md+hQLqe3/r5 wfm1XS567RN4GF+tMZjHm7qE0S7lAsN/P6lld0sCZzQL2plWf0ueZC5i0Lbjx7SJ174/qF/Fvs2 XFwA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-117-ardb@google.com> Subject: [PATCH v4 54/61] arm64: ptdump: Disregard unaddressable VA space From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071928_684114_5CC46404 X-CRM114-Status: GOOD ( 17.87 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Configurations built with support for 52-bit virtual addressing can also run on CPUs that only support 48 bits of VA space, in which case only that part of swapper_pg_dir that represents the 48-bit addressable region is relevant, and everything else is ignored by the hardware. Our software pagetable walker has little in the way of input address validation, and so it will happily start a walk from an address that is not representable by the number of paging levels that are actually active, resulting in lots of bogus output from the page table dumper unless we take care to start at a valid address. So define the start address at runtime based on vabits_actual. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/ptdump.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index f3fdbf3bb6ad..221baaf521fa 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -313,7 +313,6 @@ static void __init ptdump_initialize(void) static struct ptdump_info kernel_ptdump_info __ro_after_init = { .mm = &init_mm, - .base_addr = PAGE_OFFSET, }; void ptdump_check_wx(void) @@ -329,7 +328,7 @@ void ptdump_check_wx(void) .ptdump = { .note_page = note_page, .range = (struct ptdump_range[]) { - {PAGE_OFFSET, ~0UL}, + {_PAGE_OFFSET(vabits_actual), ~0UL}, {0, 0} } } @@ -370,6 +369,7 @@ static int __init ptdump_init(void) static struct addr_marker address_markers[ARRAY_SIZE(m)] __ro_after_init; kernel_ptdump_info.markers = memcpy(address_markers, m, sizeof(m)); + kernel_ptdump_info.base_addr = page_offset; ptdump_initialize(); ptdump_debugfs_register(&kernel_ptdump_info, "kernel_page_tables"); From patchwork Tue Sep 12 14:16:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381812 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BD478CA0EEE for ; Tue, 12 Sep 2023 14:21:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=V0gefxvpGV7cBH8gmSNzcv5smaCT0qdJ7dL0Nqy0KAo=; b=RGAOHTibJhS9Rn0R3SsxLCXcFG JsE6DUNET2DUiTupNbn4ZueO50Utad3C0w3k1xr4JCbABK92C6TUKBPxv6zsfVSWj3oViKLJ9zQQg 7RF/8RWCP77ow6NQUMiTM4v2MAHG6CPmGuv2TnzqIbB14kkzQ3+9Xf5gtNCHQeVwMgnRPDLZSKAap wQbo3RFV5ibgTlCKjA5spb2tjUw0OvU93Ks5LgAKpMSk0Nmr3jAi1ML9ZeAHavIacBU8WdBLCo+E/ ZilfByS8nNtdzktlEAsooJaSc41viqLdRPNWqFzIv0FG0aaMCM2ngLp0e19JUnntQOdUsGiJE2ZiQ PzWjwbjQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4G4-003YeI-0Q; Tue, 12 Sep 2023 14:20:52 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Em-003XXM-0L for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:37 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-3fef3606d8cso45683305e9.1 for ; Tue, 12 Sep 2023 07:19:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528369; x=1695133169; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TxbbU4Z6eJ4YLBJfIy3zQLsxMkxV5tTXMRWNfipLzKA=; b=TGfZA+oBsqS1s82hw+YQMFkHHhqzwvDUMiaik1kPx2PJ+Zc8kmPUzpu9U0l+yYy/9V biCsxD+/ZXD3utoJmWKr49IzzkWQpMa+7rUGxZYaaHeVGax/6GLjmKyk4TXWaxwkHIqZ BrBuqpBo1IfYUak0/PUn8jDtVEWktP4KLG7TnE64ACbViWkMhy0YckuP6RwVmbeVWiIB +r+7ZstDrQcREf87OcxhbQryOxVw4Yxfp1WYySJ/olVueRgaYZtJADRZm5+XfPvxA7to /AoyQrl+x5DxnSj8JzcUN/4XaneAzhztzrHNKgGbB2in+azhxB9Q7Esc5/00EG2nVMv4 8PBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528369; x=1695133169; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TxbbU4Z6eJ4YLBJfIy3zQLsxMkxV5tTXMRWNfipLzKA=; b=gA7LpEH0qhQtuWrEDp6ce0JZYT4TD+RNpCg7WylkJHH2RyYhUwu9JwI5Gi2eky/Vo9 l4PzSdCc8QfuPjMf/TEmcgJVVcfjbjbn9DwTDb+KZzJ6ePM4mcm6IKv5xX7NkHna+IQI O0lz7sfkVzxc89PhQMGqX/vXxoUl7ANCc0bLNfDi3jAgc3A/4cJPyRqyDS8h8XAK5Uyj s5Cjo/YjPmjls5Z+ElbddbLXDxVX8AnzBa8yTx/ByH86ZWAIQ+nFWnhWvpXvlGjnRXS5 vlfgZqQsYxTHc/1kzfl0MxWYn1+Mon9kEBfcGqE15tT6kprNtc2IkXSt1f0+X3NdLZWz DfzA== X-Gm-Message-State: AOJu0YzdbzxcwCmNo9VuMpkZkHRSbuRAaahHNg7xiWPlz+FK9nKhHo7+ SawNPkN6Xaj7hREH0k/mVZhPvdeNtIEETRpcZ5ptUR6R1ldBSyAjoUemgadBjdyPwl76xs1zqai mnHOPRgRWkp4cWyQN/Td+H/3TTaeFEpvYk9h3qOKYacxvSSy/JESzx8ikEnS/afVIrxRNasTQOa Q= X-Google-Smtp-Source: AGHT+IG9H3wn+iYKHzXnTZYY6brjkst5R8QfMGgUC3ZNwScPGFpu66RqMLmgPvOzzkdlAQGE/sJ8TxNp X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:f14f:0:b0:31a:dba9:ecfc with SMTP id y15-20020adff14f000000b0031adba9ecfcmr147249wro.8.1694528369122; Tue, 12 Sep 2023 07:19:29 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:45 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2891; i=ardb@kernel.org; h=from:subject; bh=EE9jwvVf5uskpfPL1qIEu4mvnVMM7BKPb1nRXoijhic=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6ALvjl2bX99J5jNQf+C5ma1rRf7bNN0sns13fR7OM nkix3C5o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAExk7h+G/3HTRAzeZaozMxpY xR5ufibfKbJIYY2vRHZZbJz709RP0xn+Oz9g8PuRJup/9dKNTQYnwi/NsP55dOrDX9rsP1JWTn1 uygwA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-118-ardb@google.com> Subject: [PATCH v4 55/61] arm64: ptdump: Deal with translation levels folded at runtime From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071932_164234_1ADF105F X-CRM114-Status: GOOD ( 20.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Currently, the ptdump code deals with folded PMD or PUD levels at build time, by omitting those levels when invoking note_page. IOW, note_page() is never invoked with level == 1 if P4Ds are folded in the build configuration. With the introduction of LPA2 support, we will defer some of these folding decisions to runtime, so let's take care of this by overriding the 'level' argument when this condition triggers. Substituting the PUD or PMD strings for "PGD" when the level in question is folded at build time is no longer necessary, and so the conditional expressions can be simplified. This also makes the indirection of the 'name' field unnecessary, so change that into a char[] array, and make the whole thing __ro_after_init. Note that the mm_p?d_folded() functions currently ignore their mm pointer arguments, but let's wire them up correctly anyway. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/ptdump.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 221baaf521fa..53925aa31051 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -48,6 +48,7 @@ struct pg_state { struct ptdump_state ptdump; struct seq_file *seq; const struct addr_marker *marker; + const struct mm_struct *mm; unsigned long start_address; int level; u64 current_prot; @@ -144,12 +145,12 @@ static const struct prot_bits pte_bits[] = { struct pg_level { const struct prot_bits *bits; - const char *name; - size_t num; + char name[4]; + int num; u64 mask; }; -static struct pg_level pg_level[] = { +static struct pg_level pg_level[] __ro_after_init = { { /* pgd */ .name = "PGD", .bits = pte_bits, @@ -159,11 +160,11 @@ static struct pg_level pg_level[] = { .bits = pte_bits, .num = ARRAY_SIZE(pte_bits), }, { /* pud */ - .name = (CONFIG_PGTABLE_LEVELS > 3) ? "PUD" : "PGD", + .name = "PUD", .bits = pte_bits, .num = ARRAY_SIZE(pte_bits), }, { /* pmd */ - .name = (CONFIG_PGTABLE_LEVELS > 2) ? "PMD" : "PGD", + .name = "PMD", .bits = pte_bits, .num = ARRAY_SIZE(pte_bits), }, { /* pte */ @@ -227,6 +228,11 @@ static void note_page(struct ptdump_state *pt_st, unsigned long addr, int level, static const char units[] = "KMGTPE"; u64 prot = 0; + /* check if the current level has been folded dynamically */ + if ((level == 1 && mm_p4d_folded(st->mm)) || + (level == 2 && mm_pud_folded(st->mm))) + level = 0; + if (level >= 0) prot = val & pg_level[level].mask; @@ -288,6 +294,7 @@ void ptdump_walk(struct seq_file *s, struct ptdump_info *info) st = (struct pg_state){ .seq = s, .marker = info->markers, + .mm = info->mm, .level = -1, .ptdump = { .note_page = note_page, From patchwork Tue Sep 12 14:16:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381814 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1BE75CA0EEF for ; Tue, 12 Sep 2023 14:21:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=QNt+9FK1Y/5I6G+oQ8XE7S0Ika1kTHyr7qbLZ92SjNU=; b=XnaXZjgnN/tJZ9Tx4lq72W3aRe 8ZTPYY686630OiTxSUchdi54I6zbVZRhnEne3ZrGTNW20siJXUI4n1QWrpl8dI3KZWvddKl0ahLki UbbTKCG0dZWt8G4u6gC2ndHiNZgP1+t1aPFFja9pB93nQtSFEQs+U9o/iJcbPkkT4FzDEgEt8pDUb GWXhRAXaLPcEB9TK5Cetz8/44GUxc+G7Z1cv3OaFbr7rP8jUrwgO9vFrMtQcNqsfKzHBmSBo93O5u 7aHOeMHBE6CzOhWEY+qAD70Ynb3qrkidasyZjqAokDNghRkMdFiwkiUmciRopmy+s9Eojl0QVQHVW B3R5N3bg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4G8-003YiD-0k; Tue, 12 Sep 2023 14:20:56 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Eo-003XYx-03 for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:45 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-58ee4df08fbso67272037b3.3 for ; Tue, 12 Sep 2023 07:19:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528372; x=1695133172; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2sdsJwIiesMTnhOVB/4fumPK21G9EJanZV4hElrFWVk=; b=WZxp1PUxVOmuvEBZbmMdS7o9z0QeO0OulRtjuNBjZvrtSNb/xgAu7D+N0VpaEqbRKw 58P1Z+RWYo7qEfS/rffK7zApvpLnNQ7IlmKvOGBfFSrEALqnbyszApZdlcyDnw0xFXEH 8Eih0AufS6qGZ01I763uYOChZhvSj0MI6fEbwcuBeeTKs7UxkmJDTeFh6ftLQrrdiGhK +QT5zVnvzEHmY1Tc954zMOBTJRLN3CadmT0Y1uBtwt9wKWDcQDya/VlvrCMFa49SHn+x c5dloiDnt9bcphDMM1c3DJjRK6/sLfc0pU2d4i8Fea4xkRb8nwJKlc0lblVp6mOFJKqz 0Zgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528372; x=1695133172; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2sdsJwIiesMTnhOVB/4fumPK21G9EJanZV4hElrFWVk=; b=BSTEcAy9jmbhRw1kQxLAjxZgI/PAd0+/hYmVYbIegO2z1ruHybrWSATwlwIvw+KUpX xZQI5dAzSWAZVdPpothkm68kHeoR/argTizGmUOhatl0sMuBkDLridH3NIt+tiZXZE59 3XhLi0yDQmGio2gOFSttiW7C5vwfGAzUjtx58/adQHh/HJR+ywSyMzXyZPuhX84asOrS 9IU7ZmWJ62rvCSc5mckdyPcEfRo/IZVGGO+uI1CImzxdmnrzBrS6hPZoExQFje+JIppM grbR9IG7pZo5GftLQgrVht5YiuPrLJZdRD4E9i5obx2VJ7cZi6oQYxB+sJASNpTNQDvk xd+g== X-Gm-Message-State: AOJu0Yx9b5N294mEG5beqKiv3pK7121xDjgi2lyzg+CmKQVFtwvnVq9m l40pM8BSUKSBXWbdF0hoEfzymdnWQcZRYD2/Miu6PsAlDwkHwIDqyYqpYsiD+qP+4+UdijcGw6O aAtR0Ttp2mKMUwTrne7yukJz+s92Ni5/nUcwWHHq57rHheQeE5SxyPpIrrWcj/TJZvOh6xK7+WD g= X-Google-Smtp-Source: AGHT+IFiInZ/Zert4boXq+8U5GwYQIyJtRisVpGLP6NK8cngEVIMI9niNbcII4LOKpEoPzad+rniwapb X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:690c:72c:b0:586:aca7:c1ee with SMTP id bt12-20020a05690c072c00b00586aca7c1eemr314873ywb.6.1694528371458; Tue, 12 Sep 2023 07:19:31 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:46 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1022; i=ardb@kernel.org; h=from:subject; bh=atPSSr2qBigDpIOZKFYcKqjzT14TPZqgrls3bMhdWNE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6GLvJedNe4t11x01rOxaUmtwl3POq49GhfX5ri4dr GtW3pToKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABOxNmBk6Dy59vzyq89Z9ZZf WB0kVegzT4jDf87b3YX23OFfZkZ9DGT4X+RqdrxebPoeljmJ/8483pTcuyg0ZN+F3E9se1+e5w9 1ZAMA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-119-ardb@google.com> Subject: [PATCH v4 56/61] arm64: kvm: avoid CONFIG_PGTABLE_LEVELS for runtime levels From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071934_113179_31C0A4FC X-CRM114-Status: GOOD ( 13.43 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel get_user_mapping_size() uses vabits_actual and CONFIG_PGTABLE_LEVELS to provide the starting point for a table walk. This is fine for LVA, as the number of translation levels is the same regardless of whether LVA is enabled. However, with LPA2, this will no longer be the case, so let's derive the number of levels from the number of VA bits directly. Signed-off-by: Ard Biesheuvel Acked-by: Marc Zyngier Acked-by: Oliver Upton --- arch/arm64/kvm/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index beee6408534d..a4c5d7f44e32 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -802,7 +802,7 @@ static int get_user_mapping_size(struct kvm *kvm, u64 addr) .pgd = (kvm_pteref_t)kvm->mm->pgd, .ia_bits = vabits_actual, .start_level = (KVM_PGTABLE_MAX_LEVELS - - CONFIG_PGTABLE_LEVELS), + ARM64_HW_PGTABLE_LEVELS(pgt.ia_bits)), .mm_ops = &kvm_user_mm_ops, }; unsigned long flags; From patchwork Tue Sep 12 14:16:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5D9B9CA0EED for ; Tue, 12 Sep 2023 14:21:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=YQXVEZg8Tqr/3JqDrnyn2hZ0M8Wh7KARHkyL48fgMHY=; b=gP2cf7lMpVHce8lEP3K4zg/lyH FF2CLVWEhnyTmCEFuVpMVED+dXvXrSQ1/pghGfAIhqQCBPMoEkM6ZoVCZlZ1397iPtZ85wNUY12+r EQw7PYWuQz47gvPsOpGo6Gzc/bN0mTOkbuimE8m+EWNaBL7NsTU887VwNqZnivk6oGJhcDqc4AsvS k1BIncYOHxgmi/AhJe3Hi3E9hm3x0qXrD5CW0MgPkg2CAPQtWP09N1EZSVsb41yVWc92wdgZdgADF oYHX1bqC8w4tQf9ijuBQQ0jMBrstmXrh/VeZmiQ1TH93NcmMtKeCUB0dQ12rgTYdlGW6cNxSq77x5 lnIXFiIQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4GE-003Yor-1Q; Tue, 12 Sep 2023 14:21:02 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Eq-003Xb1-04 for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:48 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-31400956ce8so3740213f8f.3 for ; Tue, 12 Sep 2023 07:19:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528373; x=1695133173; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QvpTwx/Nd7dA3fA7kGsBe/fNVtDfk/DIbpcHK3Zb5VQ=; b=3QjTnC13EzuedxyMaMZxaDEJ8orQwtjPrxn9Iz51N5oHXP079HJy7D3Ji1oXcvKSUb ycK/xwWUHnyVIsoP/Gkn5P3b77ctclnA1loVL5o3VNpPP2b2V1eYKyDUzoaa3HA32VY5 EbfM7hjHOXvTjNjIfdM5pDrLf/g2GezwbtwK+G2fgwC0dZHE1VAvu9ubEoqmODfa4i2f mns+TkOx+KHKDNWo5PIL57O2f/M0gVdsmcrlmjNbOUFndhbZZiw4BNqOZk+mDha6AgAC Li/4p0SewhV59Wf/Jg8mbzCHdP4YCX6X13i8qhCUimug5+7SAcsJd/pr8c1D6WE+cu+m 5HJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528373; x=1695133173; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QvpTwx/Nd7dA3fA7kGsBe/fNVtDfk/DIbpcHK3Zb5VQ=; b=kvV8/qKxSJpwppMeVjUXmZA6WHgxNy9471JBIuEs74QVNadXObcOKF3gFy1T3ZgDd2 hLSYCiplu6XDLKEnf1Hng0ceo+hvpqdj9uIQTUfChPHJNP5S2dk7xsnImxCnmSJRIlBR ZzRhNaoopGHg6FKlKeaHGft8rw9PKD5hzqzA4hcM8vvuYCZLabcklfVWtYs6oGKn4fuh vqhIuq3/vJO2D6f/rSmOA1CW7ZcpIlkbtJRvt8xE7h2oN/s9HZHd/vuhqle5e1Hqwmmf 66pLpFsMyd9HMKw/HtxbcQAhRsz3RvMat8qkF0IxJFCJ35jjAxpvhqko36wkvOfWsV6H anDw== X-Gm-Message-State: AOJu0YyIZ4scLXN8SooF84od8KiNqPfZqncF3SfIpxDIzI40GUY/S65D R0NU8zHvoevlbCFONlSGhK71l6xVP3dlzHxsqbkosX7I8gzeJFyTDGF+NXsN8uglIW1PdxQjw75 /5zUA2mOj1ZuUuLSCJzJueKHZ9BTz+JBYONronZv3gcMpe0O0fnZeJW5JqhrstHx/GhLvztcXrv o= X-Google-Smtp-Source: AGHT+IFDKViqnIq3SFDvfOjS/eTUMOqc6+LOauQcTp1xFygHgkS/RPv0PWRngp9Uf97ktS6qzGiO65Kz X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:4388:0:b0:317:6b94:b700 with SMTP id i8-20020a5d4388000000b003176b94b700mr147715wrq.9.1694528373532; Tue, 12 Sep 2023 07:19:33 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:47 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3412; i=ardb@kernel.org; h=from:subject; bh=cVjIidelPGG/9vaT00c2L45Zm3C6Gec7qO0syN6KS1E=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6JK7+w1lkdUzBGOte1nDr7ffr19z77FE/V0xEc7u+ e5Zuns6SlkYxDgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwES28TL8Lw6/sHDmVf7sucWz zUQ26K3euaJXw93svaHlRiXRn6d3bWJkWDxTgUn77dH50Rt7hC2dNxUEeEWlS7hfuh/HwnCW+7s HBwA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-120-ardb@google.com> Subject: [PATCH v4 57/61] arm64: kvm: Limit HYP VA and host S2 range to 48 bits when LPA2 is in effect From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071936_078489_4F3A4C79 X-CRM114-Status: GOOD ( 17.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The KVM code needs more work to support 5 level paging with LPA2, so for the time being, limit KVM to 48 bit addressing on 4k and 16k pagesize configurations. This can be reverted once the LPA2 support for KVM is merged. Signed-off-by: Ard Biesheuvel Acked-by: Marc Zyngier Acked-by: Oliver Upton --- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 2 ++ arch/arm64/kvm/mmu.c | 5 ++++- arch/arm64/kvm/va_layout.c | 9 +++++---- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index 9d703441278b..c20b08cf1f03 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -128,6 +128,8 @@ static void prepare_host_vtcr(void) /* The host stage 2 is id-mapped, so use parange for T0SZ */ parange = kvm_get_parange(id_aa64mmfr0_el1_sys_val); phys_shift = id_aa64mmfr0_parange_to_phys_shift(parange); + if (IS_ENABLED(CONFIG_ARM64_LPA2) && phys_shift > 48) + phys_shift = 48; // not implemented yet host_mmu.arch.vtcr = kvm_get_vtcr(id_aa64mmfr0_el1_sys_val, id_aa64mmfr1_el1_sys_val, phys_shift); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index a4c5d7f44e32..1cac302b92e4 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -800,7 +800,8 @@ static int get_user_mapping_size(struct kvm *kvm, u64 addr) { struct kvm_pgtable pgt = { .pgd = (kvm_pteref_t)kvm->mm->pgd, - .ia_bits = vabits_actual, + .ia_bits = IS_ENABLED(CONFIG_ARM64_LPA2) ? 48 + : vabits_actual, .start_level = (KVM_PGTABLE_MAX_LEVELS - ARM64_HW_PGTABLE_LEVELS(pgt.ia_bits)), .mm_ops = &kvm_user_mm_ops, @@ -1905,6 +1906,8 @@ int __init kvm_mmu_init(u32 *hyp_va_bits) idmap_bits = IDMAP_VA_BITS; kernel_bits = vabits_actual; *hyp_va_bits = max(idmap_bits, kernel_bits); + if (IS_ENABLED(CONFIG_ARM64_LPA2)) + *hyp_va_bits = 48; // LPA2 is not yet supported in KVM kvm_debug("Using %u-bit virtual addresses at EL2\n", *hyp_va_bits); kvm_debug("IDMAP page: %lx\n", hyp_idmap_start); diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c index 91b22a014610..796ffc1cc529 100644 --- a/arch/arm64/kvm/va_layout.c +++ b/arch/arm64/kvm/va_layout.c @@ -59,12 +59,13 @@ static void init_hyp_physvirt_offset(void) */ __init void kvm_compute_layout(void) { + u64 vabits = IS_ENABLED(CONFIG_ARM64_LPA2) ? 48 : vabits_actual; // not yet phys_addr_t idmap_addr = __pa_symbol(__hyp_idmap_text_start); u64 hyp_va_msb; /* Where is my RAM region? */ - hyp_va_msb = idmap_addr & BIT(vabits_actual - 1); - hyp_va_msb ^= BIT(vabits_actual - 1); + hyp_va_msb = idmap_addr & BIT(vabits - 1); + hyp_va_msb ^= BIT(vabits - 1); tag_lsb = fls64((u64)phys_to_virt(memblock_start_of_DRAM()) ^ (u64)(high_memory - 1)); @@ -72,9 +73,9 @@ __init void kvm_compute_layout(void) va_mask = GENMASK_ULL(tag_lsb - 1, 0); tag_val = hyp_va_msb; - if (IS_ENABLED(CONFIG_RANDOMIZE_BASE) && tag_lsb != (vabits_actual - 1)) { + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE) && tag_lsb != (vabits - 1)) { /* We have some free bits to insert a random tag. */ - tag_val |= get_random_long() & GENMASK_ULL(vabits_actual - 2, tag_lsb); + tag_val |= get_random_long() & GENMASK_ULL(vabits - 2, tag_lsb); } tag_val >>= tag_lsb; From patchwork Tue Sep 12 14:16:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381816 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DBFF9CA0EEC for ; Tue, 12 Sep 2023 14:21:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=TPX0kK2/ve7nsDCuWz6+1Riol0bIrTOCqUA8xVhB4ws=; b=smcDpmJRodJRSONAjvAypCvtUc BAzMYwkiYLmhx9vK6xJK+cDvU/02nXRFCYI+vgX62tU7yvs7wsyXUS8acKr0WEYxI96QG7/4JuIsT ZwF3fcxqX7FipKp+fCgtZBB3Ce70PHLs7u5j747cmzubHGNqcG0oNYTZa/J26PfBhH7rtzMqOP/fJ 595DOA8k2imFfxLqpr6SNKVz+5IFUxKvawYUMFE/kCMLrs4rAv7shZOHO6iZLKM6fMeaMZJetW/4n bkBec3LKXhaCJ7bQvBKHRG88abaHdzmYwZQEYPqJDCxStLvQHqtHOe7UxxUA7n/rwdMVzJoqez21f kzi9WoOA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4GC-003Yls-03; Tue, 12 Sep 2023 14:21:00 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Es-003Xcu-0H for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:49 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-401bdff6bc5so41258965e9.1 for ; Tue, 12 Sep 2023 07:19:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528376; x=1695133176; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lQDaFzj41m3HJVdK9i3GHdbGuRCOufrl3u/xYqy13f4=; b=pH/kuh13kfhbLRs+xGDFSWke1qebIlwFPb4Kh2YySnqoPVykwb+brb9IxpN6isGmh4 RR8uAA3zkida1UUW83PuMR4U75ExTVQTW91v1u2QHlHNfUMdKcn0Rfna7AB/nAKD6MfC u9XzvLgz7QG80TUQd87pran+ueriFG/qQJgjScp580s0TVmoUDFWyWuP11Q04qgwl6ON QoysK/DgQ/l+yRTRKlMV6rOda8eYaAZ/dY3r5a5IlU4WgwETccIzY1E8OEHooLvkqrAI 0XxKQA8QNGQ1+ghGhHPHhNX70PbU2MuNeQSxtL7zTZ9fMqyA+j9aSoUyDH60da0Qd41/ UVTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528376; x=1695133176; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lQDaFzj41m3HJVdK9i3GHdbGuRCOufrl3u/xYqy13f4=; b=fdGkiG07VZcmNqg/N+FWvu0iyDu6b3a2mJrtr/iHuNiw/p7sZZwPtENCkfyk/CLEXM ty0r6vxK2nfch+8M65+ydotC2LkRPoik2NN26ce1Ubsb1eviitxjqrM7er8qFOIMYVU4 ZfeSvE6lZGi+kkeHU92fET0i1FBW66HKVT+ts+72fism29B/2G6COAoVX1+eSW6RCiIa 5aY3LLZk+o67/HXeMZ4JgBKBMVBqSCMxJCQLwMK3QbjkFJb5tITo4nIRMIziKsGp8ZHU UddV2wutFNsC5KOudmQ00kzFqNQgdVZUGWz8rvruOarYDJQFAVcx5KCmrnX98vhqvZSl PAjA== X-Gm-Message-State: AOJu0YwZpPRS08RfrrXRWScgK/agj5bD+huNYqpcvX1loq6pCuIwhg6P qbFuATL7Xc9qVH8esYHtLuoFQ6jtzt9bvJ1KYYfcRIdDuv6fHX0F5vX8vFhS1d6Pywzc4Im0Uk/ Lh6CXtLZgiYzooVEgstw6cJux5D64/wGXw1rPlEfCKFeBtMuXwCIAQxscWSYiEFPtEgztTzqsHj I= X-Google-Smtp-Source: AGHT+IFpO7qtNsV3oo6V5qZrFu46Sors3a0TMQ24EvLLqfiMdS+lEqFyh2uoiR0XBjGqGKBYLY6NnrRJ X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a7b:cd96:0:b0:3fb:bcec:72da with SMTP id y22-20020a7bcd96000000b003fbbcec72damr214359wmj.5.1694528375915; Tue, 12 Sep 2023 07:19:35 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:48 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5018; i=ardb@kernel.org; h=from:subject; bh=/wvqAYISaVfJ/wHAu7n4oKL5WYHZ7g+M4SPjPO635Ds=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6LKo857Wd4qKtqn9Tx6yLJo6I5OlyWUWg8uNGOaO9 26P/up1lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgInY7GD4n/m7blUcW9tN91v7 Y5V2b/FXfXvkWrDu+2wb98n/dPkYPjIytNhK8Xy66CWYGKEUv7u5pPGJV3PU3AnLTj++a8we+qm bFwA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-121-ardb@google.com> Subject: [PATCH v4 58/61] arm64: Enable 52-bit virtual addressing for 4k and 16k granule configs From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071938_147941_EC897A30 X-CRM114-Status: GOOD ( 17.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Update Kconfig to permit 4k and 16k granule configurations to be built with 52-bit virtual addressing, now that all the prerequisites are in place. While at it, update the feature description so it matches on the appropriate feature bits depending on the page size. For simplicity, let's just keep ARM64_HAS_VA52 as the feature name. Note that LPA2 based 52-bit virtual addressing requires 52-bit physical addressing support to be enabled as well, as programming TCR.TxSZ to values below 16 is not allowed unless TCR.DS is set, which is what activates the 52-bit physical addressing support. While supporting the converse (52-bit physical addressing without 52-bit virtual addressing) would be possible in principle, let's keep things simple, by only allowing these features to be enabled at the same time. Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 17 ++++++++------- arch/arm64/kernel/cpufeature.c | 22 ++++++++++++++++---- 2 files changed, 28 insertions(+), 11 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 3a8d40a2adb7..24ce9be834c1 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -366,7 +366,9 @@ config PGTABLE_LEVELS default 3 if ARM64_64K_PAGES && (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) default 3 if ARM64_4K_PAGES && ARM64_VA_BITS_39 default 3 if ARM64_16K_PAGES && ARM64_VA_BITS_47 + default 4 if ARM64_16K_PAGES && (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) default 4 if !ARM64_64K_PAGES && ARM64_VA_BITS_48 + default 5 if ARM64_4K_PAGES && ARM64_VA_BITS_52 config ARCH_SUPPORTS_UPROBES def_bool y @@ -394,13 +396,13 @@ config BUILTIN_RETURN_ADDRESS_STRIPS_PAC config KASAN_SHADOW_OFFSET hex depends on KASAN_GENERIC || KASAN_SW_TAGS - default 0xdfff800000000000 if (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) && !KASAN_SW_TAGS - default 0xdfffc00000000000 if ARM64_VA_BITS_47 && !KASAN_SW_TAGS + default 0xdfff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && !KASAN_SW_TAGS + default 0xdfffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && !KASAN_SW_TAGS default 0xdffffe0000000000 if ARM64_VA_BITS_42 && !KASAN_SW_TAGS default 0xdfffffc000000000 if ARM64_VA_BITS_39 && !KASAN_SW_TAGS default 0xdffffff800000000 if ARM64_VA_BITS_36 && !KASAN_SW_TAGS - default 0xefff800000000000 if (ARM64_VA_BITS_48 || ARM64_VA_BITS_52) && KASAN_SW_TAGS - default 0xefffc00000000000 if ARM64_VA_BITS_47 && KASAN_SW_TAGS + default 0xefff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && KASAN_SW_TAGS + default 0xefffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && KASAN_SW_TAGS default 0xeffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS default 0xefffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS @@ -1277,7 +1279,7 @@ config ARM64_VA_BITS_48 config ARM64_VA_BITS_52 bool "52-bit" - depends on ARM64_64K_PAGES && (ARM64_PAN || !ARM64_SW_TTBR0_PAN) + depends on ARM64_PAN || !ARM64_SW_TTBR0_PAN help Enable 52-bit virtual addressing for userspace when explicitly requested via a hint to mmap(). The kernel will also use 52-bit @@ -1324,10 +1326,11 @@ choice config ARM64_PA_BITS_48 bool "48-bit" + depends on ARM64_64K_PAGES || !ARM64_VA_BITS_52 config ARM64_PA_BITS_52 - bool "52-bit (ARMv8.2)" - depends on ARM64_64K_PAGES + bool "52-bit" + depends on ARM64_64K_PAGES || ARM64_VA_BITS_52 depends on ARM64_PAN || !ARM64_SW_TTBR0_PAN help Enable support for a 52-bit physical address space, introduced as diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 778f42cfcf90..8f5b83f5417b 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2689,15 +2689,29 @@ static const struct arm64_cpu_capabilities arm64_features[] = { }, #ifdef CONFIG_ARM64_VA_BITS_52 { - .desc = "52-bit Virtual Addressing (LVA)", .capability = ARM64_HAS_VA52, .type = ARM64_CPUCAP_BOOT_CPU_FEATURE, - .sys_reg = SYS_ID_AA64MMFR2_EL1, - .sign = FTR_UNSIGNED, + .matches = has_cpuid_feature, .field_width = 4, +#ifdef CONFIG_ARM64_64K_PAGES + .desc = "52-bit Virtual Addressing (LVA)", + .sign = FTR_SIGNED, + .sys_reg = SYS_ID_AA64MMFR2_EL1, .field_pos = ID_AA64MMFR2_EL1_VARange_SHIFT, - .matches = has_cpuid_feature, .min_field_value = ID_AA64MMFR2_EL1_VARange_52, +#else + .desc = "52-bit Virtual Addressing (LPA2)", + .sys_reg = SYS_ID_AA64MMFR0_EL1, +#ifdef CONFIG_ARM64_4K_PAGES + .sign = FTR_SIGNED, + .field_pos = ID_AA64MMFR0_EL1_TGRAN4_SHIFT, + .min_field_value = ID_AA64MMFR0_EL1_TGRAN4_52_BIT, +#else + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR0_EL1_TGRAN16_SHIFT, + .min_field_value = ID_AA64MMFR0_EL1_TGRAN16_52_BIT, +#endif +#endif }, #endif {}, From patchwork Tue Sep 12 14:16:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381868 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A9C78CA0EF6 for ; Tue, 12 Sep 2023 15:30:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=9NFSQjNq8Lg9OGrF3rj+n4XGoGDNQtX+KvnXxv00YL8=; b=v73NwiMDz7myZ2Cgx5znvfxpp9 wxvBabtL0PnD08lvDnbAUQIZ9azhvuCDm+c7M2B/DK3ejl+w0ETZ4W5XQYUVp2hDuzktuVrTgUcgn DhHOUxKuIO4XnfixMVSQ+5XDlw1tn8c0Wyiq9yhjP2E3TolA3A2jj3DSMt/69Qg2Tp84JkokW/bLb CfdcoDzF06r/ZyN3Lq7wsVFDgw3LOzvyMb/p+6TRtDUWA0Cb+aQfSy8vftPVZh49oLhuntol7lyT4 MeEzxnA0Ut47JS2Zd7TTpxYvcFndUieLROywQ864j3A4daPtSeo8effp4fNipylFMZ0JScP0meqv9 tZpXi6LQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg5LS-003ii8-1A; Tue, 12 Sep 2023 15:30:30 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Eu-003Xez-1U for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:51 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-59b6083fa00so43655077b3.0 for ; Tue, 12 Sep 2023 07:19:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528378; x=1695133178; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7IqPiE8iKNadvphyB/GGPEIly6u599RMMe1HgznquLA=; b=JMNclmOio33x8B6UynZ7VfYd5qTM0qeUjJA4VpP6WFARXE8nyY5RSUrLC9Qj0VDMFQ QH/EEx20QDolO19tafdqt9dpugFDDTPHPVracfMcytaJNRfhp0C48ca7A9ZcGBSsjuPO /qPv+Zbf06RzvwfkFdSxeRtUPVsG4wlN9eEdbFJ01P5XRsKt+qbfxPKAYs5OVN99CSsy Zznq9yQBV/wi1JQnBcE3syzolT5U7jm/EGdXBt9UzXPZ5fUnajCNC/lbB/XipC2GewOu SSRoy/Gt/aJWLuWyQyDrAWK4JePAIg/9AbTbtAhSejnHYCSGRRZexI8KTOnrvF/MtQqR 0wtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528378; x=1695133178; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7IqPiE8iKNadvphyB/GGPEIly6u599RMMe1HgznquLA=; b=ul/GS9VvLV2/YJD3HiopwBqXm8G8qTsSm5qjeH4WG2WtP2bmfVySrN2SnCmnrDZjdV jEW7P2laMKljrBSfIXVTLJjxzrzB/ZeRbVMy+jjdkbEYC+j4cWrQdQg9mN7DXAsgtmYu NLVVGg/tQuyaXhIIwj1sM7CrlbArvoiaaxNW611cBH6J/a/Kn2U1BTxVYIg89nbaA6Qs crE+xxenDsbOdwdyZaibwWf0jMOYpfn0bQTCBxrQeaVHad3u2Q+5WOU5Glo/TWe/1U2m 6umpicofZAYxGDZDG82r/T8QM8tQGbX4c6rCmjwwvXPh866NNq/Q19lfM6/Oc1xIjcx4 5g7A== X-Gm-Message-State: AOJu0YwJQSyJj1b1nRL/iVpntZUV3np9g/AtfhZf+weRhWIBuXl9A2fM SpDWpaBGRVqrA9ncPx7HO4Y7OORgcAHeavnoN3j6V0ZBIcwIC+gCDNgB0stFH3/zun+ShlIeik9 XCG0WDOeXXJgg+3bUGCy35nb+kQeVqYt2niApCR24WOM1uynyXKH2QueWjELSMk3+O43C664Ti7 U= X-Google-Smtp-Source: AGHT+IEkSsi/PZBMaUolbLMjWPpRa5SFB5GG08NSljWbgAYwmJ8Sr8bIu7+zVPIqmRwtTQrVZQ47ZSGB X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:7642:0:b0:586:5d03:67c8 with SMTP id j2-20020a817642000000b005865d0367c8mr360572ywk.3.1694528378322; Tue, 12 Sep 2023 07:19:38 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:49 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=959; i=ardb@kernel.org; h=from:subject; bh=RDIusN2O05VpHUK3UZT9bgZAs17M4uMCZj7ybEF9c+w=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6IpPL/N+puj83OJi+3nKIlKcaXVvtu2I8cjbm3WtK 26L1fKOUhYGMQ4GWTFFFoHZf9/tPD1RqtZ5lizMHFYmkCEMXJwCMJGkBob/NS//pqoc+57y8owC q8jBSyma4geLW69JLS1nEZhjG+dlxfDPfObkc5yLGqdNF962nefeBY73AjtqNj85bhu/IyQsZ9k lLgA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-122-ardb@google.com> Subject: [PATCH v4 59/61] arm64: defconfig: Enable LPA2 support From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071940_612630_8DC0F074 X-CRM114-Status: GOOD ( 13.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel We typically enable support in defconfig for all architectural features for which we can detect at runtime if the hardware actually supports them. Now that we have implemented support for LPA2 based 52-bit virtual addressing in a way that should not impact 48-bit operation on non-LPA2 CPU, we can do the same, and enable 52-bit virtual addressing by default. Signed-off-by: Ard Biesheuvel --- arch/arm64/configs/defconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig index 5315789f4868..214db88934b8 100644 --- a/arch/arm64/configs/defconfig +++ b/arch/arm64/configs/defconfig @@ -73,7 +73,7 @@ CONFIG_ARCH_VEXPRESS=y CONFIG_ARCH_VISCONTI=y CONFIG_ARCH_XGENE=y CONFIG_ARCH_ZYNQMP=y -CONFIG_ARM64_VA_BITS_48=y +CONFIG_ARM64_VA_BITS_52=y CONFIG_SCHED_MC=y CONFIG_SCHED_SMT=y CONFIG_NUMA=y From patchwork Tue Sep 12 14:16:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381869 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 271C4CA0EF3 for ; Tue, 12 Sep 2023 15:30:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=f0RbQ2bUj5DgDOTZGM5h84iy5WWbSvXK6KFJ+N0sVWE=; b=xDg4zmUUmr/9Sshw7pjxiHVVLZ yWbDIs94atOG8SkmJ22uoC5zW1NlD1DR0cPGhdfmSolNzsA7znzA1pLL72eOHKTQunSrpx4S8Ce+E j0JsyD8TtO85nXiyr9+UzuPkyg5RW680NvxWOIWkCQMVhMMJD0yLk+SGIvZsmj1gzYfWl3ho46Wsu kUddQWRz8llJ3l83yXY7RjVGtd+OLyZ2rfWr9mdRPldOOow6LhBh8CVhkRJHEFyI3bNh5xEplSfv/ dXinFA6YRvjycOFg7KPIJ33ip5kUp80ql1MVaL4J83EtQcQwZpk62uP9fO76jK/XklF0p6LhdHG/O Njwn1fHg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg5LT-003iih-0M; Tue, 12 Sep 2023 15:30:31 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Ex-003Xgj-28 for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:54 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-59b6083fa00so43655597b3.0 for ; Tue, 12 Sep 2023 07:19:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528381; x=1695133181; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9L8N3hd/WX9xSYeSPRyn0Xezc0wg7YRXnOdzcJZR+T8=; b=K9ItLRCUE3SOb/evBjGjIgJvLrYWh4hqPozL7l/i4Y9GALmOvwHb69QK069N+awPrP P8x9Z/HZ3IfvJvVRyRzG23JbKYvBxy9qvldEPIBnlVK7/YU0NsvLrmIaLuWTlSKR79UQ CoBFEZy3ON+kiQc5Q9elLO3esV1iNkKfQ3OYf2U2iih2WTOSpqECYpoVLRoP7iyvwi6B KgyejD7PG/6W5WfhHOAUveESJodvlXJj+IAblwPwnEItB6gRQN+TKEpeEzxtN534psBI TaMjhY9zG4++h+DHxDa+iJuULEAVjehFWT6nCdcjOiXb8iE/qrb1WLL/215f8XQeRd5O mT5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528381; x=1695133181; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9L8N3hd/WX9xSYeSPRyn0Xezc0wg7YRXnOdzcJZR+T8=; b=Ef5cvJZQGjB/5anBviba0zPAOgg8j6jPG/GTczvBk3RycL1FEIuAU6C4052g2Icp0I oIVVcP1+DeoTdvGRS0fYHadlj9RHDQlKdS4/MUcSO/S+x0bP19KS/2rvVGHKn2aP7lGr hL+m3YsQGpL2+lIvRR5i+PUK7h6bLoHRn+j+yza244DJ00a6Hw+0ZqlOoOX/1/3NyUme I3eDBG46IMcOeQbQPJVFXtJoEnWOCiPMyd4WoOa48pH0cup6xom4yhy5Kz5I1F7bCLcG XUSmo7UsUJZddOYP/zKm3CarJUhnKdrUJwDOHc3bdHYvOOYVUTXQLhv11BsV/0DQQMbj DMpA== X-Gm-Message-State: AOJu0YyIMLuGoHRQhd7LuOUh03QmZbCouVVa61PzNPiKije9DFrIMDo7 PGcHbO8dOoPxDAsnLzjZ66u80OsZabozvWKJUbdgCFM15qDSxEDHeek97t1IXB+yo7iztwmjBxm NZ8pp8YI1RwhGriLEjutrePfc3UBSO65dHg8lu4NHX0dNoQ5zVgTA3q7Q+Z6dR1L/gze7HzW8C+ c= X-Google-Smtp-Source: AGHT+IEU90q9RCpyjWB1Fqf7BdClyO67zAY2ugSFebkaTf3MYrcIWqJg/DAgsEHz4RPA7u/VZVZyAUAZ X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:b706:0:b0:58c:74ec:339b with SMTP id v6-20020a81b706000000b0058c74ec339bmr297303ywh.1.1694528380773; Tue, 12 Sep 2023 07:19:40 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:50 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1750; i=ardb@kernel.org; h=from:subject; bh=pu/U7f4DkZFTkoG+/EW/geTfpkkMEHP2k/cuZkS2X7Y=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6KoAT06TaNtSMZY+MwtF32OVUhn891yvXZVoD813z HP/vqSjlIVBjINBVkyRRWD233c7T0+UqnWeJQszh5UJZAgDF6cATIS5mpFhzcN7ak/TZjHx1vg9 PTlTJHRdbsAhq7nmT+6uDZhw4k7pY4Z/5v82/sjbZ3Pjx9l1ZbE12wVDxWT/5i9uWHcnKt7/1cs sdgA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-123-ardb@google.com> Subject: [PATCH v4 60/61] mm: add arch hook to validate mmap() prot flags From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071943_769681_25AB3CB3 X-CRM114-Status: GOOD ( 14.16 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add a hook to permit architectures to perform validation on the prot flags passed to mmap(), like arch_validate_prot() does for mprotect(). This will be used by arm64 to reject PROT_WRITE+PROT_EXEC mappings on configurations that run with WXN enabled. Reviewed-by: Kees Cook Signed-off-by: Ard Biesheuvel --- include/linux/mman.h | 15 +++++++++++++++ mm/mmap.c | 3 +++ 2 files changed, 18 insertions(+) diff --git a/include/linux/mman.h b/include/linux/mman.h index 40d94411d492..f5d8c714b4bd 100644 --- a/include/linux/mman.h +++ b/include/linux/mman.h @@ -124,6 +124,21 @@ static inline bool arch_validate_flags(unsigned long flags) #define arch_validate_flags arch_validate_flags #endif +#ifndef arch_validate_mmap_prot +/* + * This is called from mmap(), which ignores unknown prot bits so the default + * is to accept anything. + * + * Returns true if the prot flags are valid + */ +static inline bool arch_validate_mmap_prot(unsigned long prot, + unsigned long addr) +{ + return true; +} +#define arch_validate_mmap_prot arch_validate_mmap_prot +#endif + /* * Optimisation macro. It is equivalent to: * (x & bit1) ? bit2 : 0 diff --git a/mm/mmap.c b/mm/mmap.c index b56a7f0c9f85..8dcd3f87063b 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1204,6 +1204,9 @@ unsigned long do_mmap(struct file *file, unsigned long addr, if (!(file && path_noexec(&file->f_path))) prot |= PROT_EXEC; + if (!arch_validate_mmap_prot(prot, addr)) + return -EACCES; + /* force arch specific MAP_FIXED handling in get_unmapped_area */ if (flags & MAP_FIXED_NOREPLACE) flags |= MAP_FIXED; From patchwork Tue Sep 12 14:16:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381870 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 732D5CA0EEB for ; Tue, 12 Sep 2023 15:30:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=i1EFZzlYLO65D0krAzuGuV13aBicmfUFTbxNshIOcEY=; b=X7Q4Li30pHi6W5Q0ErF2LaxLUM qpg7RaE3VPZy6VgHmPegeLBTA1mGsXlhKd+I/1NGcE+mIPRYpmeVOHIq21HdFhPsV2rRzotNl9kvO iDqr4KL1iiCp5wJEIGRfxt1rOv7g4d57vBdC0I8BjL/eemrubqgSJpG1XS2JMJTB8mLDk+a6wtih6 PHq6VVDxCgVTd6f1w8SCHIr8izG2m47w9k2IffJleyy3pzVfrrGqQwSSFafnN+rHRC3/XAVmB6Dco tAhfcH9V6oS8VWxYybPm8ZP8/4t0URSELHlUQu2RUgpkE9/8poqwe/5ETRApL/vDWszu7LgeedAjZ 8DFSuWRw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg5LT-003ij6-29; Tue, 12 Sep 2023 15:30:31 +0000 Received: from mail-ej1-x64a.google.com ([2a00:1450:4864:20::64a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Ez-003XiX-35 for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:58 +0000 Received: by mail-ej1-x64a.google.com with SMTP id a640c23a62f3a-94a355c9028so373274466b.3 for ; Tue, 12 Sep 2023 07:19:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528383; x=1695133183; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pP9ux+t7ybPRTLPm12BO/OZQWRNexAHVXS3UA5UcNZg=; b=B5QnGlY6YVopf0GIrBLlMxfTHi8LgzUNTqzn63Cudhp5kZ7N+dwNnFmf8IFKE5QbIU kIAbnBncHC9unL5Ifn/jJ13/q6hJ9re3WUq4Og2MEjrai9VwCl7ELiIGSPZmtHCByoDh XqxgXbJim6nbgV+whajOy4Q+B2t8ycO9oQ8raPgezvfYw6lYUQmtePlXzzY+IGmgczA2 VFmIQpu6ta44CuWJsuT5b07K/J/G8BqOEE+7ENsX7vU4alQI014b++MZ8ZF+kQeq055n 9wJ8dEKOeIg+bAX8kMmPwzSFDpzvRu3F5BPpsdS0EVjObXW0N5CdQAOgVDl2gZ3pG0oC SvXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528383; x=1695133183; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pP9ux+t7ybPRTLPm12BO/OZQWRNexAHVXS3UA5UcNZg=; b=FfwG+EtP+rOu6qNFRGdXUJUhAsxXUQESrQILI8ZsXxiG+nC2uKokSkHKdNx3yo8NuN yl/CaEyk7rmR6LGGGqag77TI3EYj7KKaGk7sD3xFPosA5D0qmflGym6Gl8v0dP2CE6k6 Q2i69gZ50771zlUEgk5s/Arq623SFFtAMRecKvR4qRoj6LJF+VaB7LzInbm7MbRFnt2l eY/zB4Ya2nmnKAquEz1hrei5/wgov5lGwXgNow7WxyBWGDA65gIYRZagtzTEiGvjluxi nZRpH/JudB+wPaJ3P52M82d8nU+b8g/zhl+OIgZ7tGtYETHVnrcIzUKf3+gRwLK+hTyg gsIg== X-Gm-Message-State: AOJu0Yxk0KKui+l4NrGldCwCafzB++Nd2yjJur1AC2vcoAT9c+2G35o0 /1hbS6l+mpwJIPJp6LwgK6h/cr8WNr/8YCl8uAbC/YwAh1wko27h5xQ36diICZStJkt1kDOHebr ZWMj+thlmf6mYCYpP3dDYh2C50j+ADC0UEkvgkADpL+/ltFYNM6cjV6P/h25CQQTXA7N3IHyOT5 U= X-Google-Smtp-Source: AGHT+IFc5FzZka06b1EqPOj2yNTfErdOLl8goiBxj1OycXp8ezZfFNShyD6O9YFAj2amLuNDNkYiwgbA X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a17:907:621c:b0:9a1:f5c4:acb2 with SMTP id ms28-20020a170907621c00b009a1f5c4acb2mr89201ejc.4.1694528383042; Tue, 12 Sep 2023 07:19:43 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:51 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10057; i=ardb@kernel.org; h=from:subject; bh=Pkejha3L+hehiRFM85wJvx2t6LHOX9hbW4dhnDPZzFI=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6Nq1sDU6dY5SUj7/Fj7WbZQ7XnpvzT2/FefLg01uv g54Oq2zo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzkyHeGf3Z3fFa8vZwrIFKx +tuSlRMPHXP4/HrOoh8vDLz3S6nu+V3OyPD29a30pX6nGPiFyz1tM1aZVX2btrDMOq2Jk4f15p4 /otwA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-124-ardb@google.com> Subject: [PATCH v4 61/61] arm64: mm: add support for WXN memory translation attribute From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071946_026472_2388D218 X-CRM114-Status: GOOD ( 30.23 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The AArch64 virtual memory system supports a global WXN control, which can be enabled to make all writable mappings implicitly no-exec. This is a useful hardening feature, as it prevents mistakes in managing page table permissions from being exploited to attack the system. When enabled at EL1, the restrictions apply to both EL1 and EL0. EL1 is completely under our control, and has been cleaned up to allow WXN to be enabled from boot onwards. EL0 is not under our control, but given that widely deployed security features such as selinux or PaX already limit the ability of user space to create mappings that are writable and executable at the same time, the impact of enabling this for EL0 is expected to be limited. (For this reason, common user space libraries that have a legitimate need for manipulating executable code already carry fallbacks such as [0].) If enabled at compile time, the feature can still be disabled at boot if needed, by passing arm64.nowxn on the kernel command line. [0] https://github.com/libffi/libffi/blob/master/src/closures.c#L440 Signed-off-by: Ard Biesheuvel Reviewed-by: Kees Cook --- arch/arm64/Kconfig | 11 ++++++ arch/arm64/include/asm/cpufeature.h | 10 ++++++ arch/arm64/include/asm/mman.h | 36 ++++++++++++++++++++ arch/arm64/include/asm/mmu_context.h | 30 +++++++++++++++- arch/arm64/kernel/pi/idreg-override.c | 4 ++- arch/arm64/kernel/pi/map_kernel.c | 24 +++++++++++++ arch/arm64/mm/proc.S | 6 ++++ 7 files changed, 119 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 24ce9be834c1..3aa50f42ed5c 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1572,6 +1572,17 @@ config RODATA_FULL_DEFAULT_ENABLED This requires the linear region to be mapped down to pages, which may adversely affect performance in some cases. +config ARM64_WXN + bool "Enable WXN attribute so all writable mappings are non-exec" + help + Set the WXN bit in the SCTLR system register so that all writable + mappings are treated as if the PXN/UXN bit is set as well. + If this is set to Y, it can still be disabled at runtime by + passing 'arm64.nowxn' on the kernel command line. + + This should only be set if no software needs to be supported that + relies on being able to execute from writable mappings. + config ARM64_SW_TTBR0_PAN bool "Emulate Privileged Access Never using TTBR0_EL1 switching" help diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 5eed4a12c625..484ce8d1d3d4 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -18,6 +18,7 @@ #define ARM64_SW_FEATURE_OVERRIDE_NOKASLR 0 #define ARM64_SW_FEATURE_OVERRIDE_HVHE 4 #define ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF 8 +#define ARM64_SW_FEATURE_OVERRIDE_NOWXN 12 #ifndef __ASSEMBLY__ @@ -932,6 +933,15 @@ static inline bool kaslr_disabled_cmdline(void) return false; } +static inline bool arm64_wxn_enabled(void) +{ + if (!IS_ENABLED(CONFIG_ARM64_WXN) || + cpuid_feature_extract_unsigned_field(arm64_sw_feature_override.val, + ARM64_SW_FEATURE_OVERRIDE_NOWXN)) + return false; + return true; +} + u32 get_kvm_ipa_limit(void); void dump_cpu_features(void); diff --git a/arch/arm64/include/asm/mman.h b/arch/arm64/include/asm/mman.h index 5966ee4a6154..6d4940342ba7 100644 --- a/arch/arm64/include/asm/mman.h +++ b/arch/arm64/include/asm/mman.h @@ -35,11 +35,40 @@ static inline unsigned long arch_calc_vm_flag_bits(unsigned long flags) } #define arch_calc_vm_flag_bits(flags) arch_calc_vm_flag_bits(flags) +static inline bool arm64_check_wx_prot(unsigned long prot, + struct task_struct *tsk) +{ + /* + * When we are running with SCTLR_ELx.WXN==1, writable mappings are + * implicitly non-executable. This means we should reject such mappings + * when user space attempts to create them using mmap() or mprotect(). + */ + if (arm64_wxn_enabled() && + ((prot & (PROT_WRITE | PROT_EXEC)) == (PROT_WRITE | PROT_EXEC))) { + /* + * User space libraries such as libffi carry elaborate + * heuristics to decide whether it is worth it to even attempt + * to create writable executable mappings, as PaX or selinux + * enabled systems will outright reject it. They will usually + * fall back to something else (e.g., two separate shared + * mmap()s of a temporary file) on failure. + */ + pr_info_ratelimited( + "process %s (%d) attempted to create PROT_WRITE+PROT_EXEC mapping\n", + tsk->comm, tsk->pid); + return false; + } + return true; +} + static inline bool arch_validate_prot(unsigned long prot, unsigned long addr __always_unused) { unsigned long supported = PROT_READ | PROT_WRITE | PROT_EXEC | PROT_SEM; + if (!arm64_check_wx_prot(prot, current)) + return false; + if (system_supports_bti()) supported |= PROT_BTI; @@ -50,6 +79,13 @@ static inline bool arch_validate_prot(unsigned long prot, } #define arch_validate_prot(prot, addr) arch_validate_prot(prot, addr) +static inline bool arch_validate_mmap_prot(unsigned long prot, + unsigned long addr) +{ + return arm64_check_wx_prot(prot, current); +} +#define arch_validate_mmap_prot arch_validate_mmap_prot + static inline bool arch_validate_flags(unsigned long vm_flags) { if (!system_supports_mte()) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 4eeb460d7ed6..35cac8ae7d54 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -20,13 +20,41 @@ #include #include #include -#include #include #include #include extern bool rodata_full; +static inline int arch_dup_mmap(struct mm_struct *oldmm, + struct mm_struct *mm) +{ + return 0; +} + +static inline void arch_exit_mmap(struct mm_struct *mm) +{ +} + +static inline void arch_unmap(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ +} + +static inline bool arch_vma_access_permitted(struct vm_area_struct *vma, + bool write, bool execute, bool foreign) +{ + if (IS_ENABLED(CONFIG_ARM64_WXN) && execute && + (vma->vm_flags & (VM_WRITE | VM_EXEC)) == (VM_WRITE | VM_EXEC)) { + pr_warn_ratelimited( + "process %s (%d) attempted to execute from writable memory\n", + current->comm, current->pid); + /* disallow unless the nowxn override is set */ + return !arm64_wxn_enabled(); + } + return true; +} + static inline void contextidr_thread_switch(struct task_struct *next) { if (!IS_ENABLED(CONFIG_PID_IN_CONTEXTIDR)) diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index 062aeae68936..1912aae12de3 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -193,6 +193,7 @@ static const struct ftr_set_desc sw_features __prel64_initconst = { FIELD("nokaslr", ARM64_SW_FEATURE_OVERRIDE_NOKASLR, NULL), FIELD("hvhe", ARM64_SW_FEATURE_OVERRIDE_HVHE, hvhe_filter), FIELD("rodataoff", ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF, NULL), + FIELD("nowxn", ARM64_SW_FEATURE_OVERRIDE_NOWXN, NULL), {} }, }; @@ -227,8 +228,9 @@ static const struct { { "arm64.nomops", "id_aa64isar2.mops=0" }, { "arm64.nomte", "id_aa64pfr1.mte=0" }, { "nokaslr", "arm64_sw.nokaslr=1" }, - { "rodata=off", "arm64_sw.rodataoff=1" }, + { "rodata=off", "arm64_sw.rodataoff=1 arm64_sw.nowxn=1" }, { "arm64.nolva", "id_aa64mmfr2.varange=0" }, + { "arm64.nowxn", "arm64_sw.nowxn=1" }, }; static int __init parse_hexdigit(const char *p, u64 *v) diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index 591394befa5f..be7caf07bfa7 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -133,6 +133,25 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) idmap_cpu_replace_ttbr1(swapper_pg_dir); } +static void noinline __section(".idmap.text") disable_wxn(void) +{ + u64 sctlr = read_sysreg(sctlr_el1) & ~SCTLR_ELx_WXN; + + /* + * We cannot safely clear the WXN bit while the MMU and caches are on, + * so turn the MMU off, flush the TLBs and turn it on again but with + * the WXN bit cleared this time. + */ + asm(" msr sctlr_el1, %0 ;" + " isb ;" + " tlbi vmalle1 ;" + " dsb nsh ;" + " isb ;" + " msr sctlr_el1, %1 ;" + " isb ;" + :: "r"(sctlr & ~SCTLR_ELx_M), "r"(sctlr)); +} + static void noinline __section(".idmap.text") set_ttbr0_for_lpa2(u64 ttbr) { u64 sctlr = read_sysreg(sctlr_el1); @@ -230,6 +249,11 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) if (va_bits > VA_BITS_MIN) sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(va_bits)); + if (IS_ENABLED(CONFIG_ARM64_WXN) && + cpuid_feature_extract_unsigned_field(arm64_sw_feature_override.val, + ARM64_SW_FEATURE_OVERRIDE_NOWXN)) + disable_wxn(); + /* * The virtual KASLR displacement modulo 2MiB is decided by the * physical placement of the image, as otherwise, we might not be able diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index f32e4b087ef8..cda34e96f5f5 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -547,6 +547,12 @@ alternative_else_nop_endif * Prepare SCTLR */ mov_q x0, INIT_SCTLR_EL1_MMU_ON +#ifdef CONFIG_ARM64_WXN + ldr_l x1, arm64_sw_feature_override + FTR_OVR_VAL_OFFSET + tst x1, #0xf << ARM64_SW_FEATURE_OVERRIDE_NOWXN + orr x1, x0, #SCTLR_ELx_WXN + csel x0, x0, x1, ne +#endif ret // return to head.S .unreq mair