From patchwork Fri Nov 24 10:19:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13467456 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 60EF1C61D97 for ; Fri, 24 Nov 2023 10:23:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=48Cnm8/pcFfA8uVfo2aTiU1WJdvPL2jxZUskUmS6Raw=; b=LD21ApuQ9HoWv/shs8xL/IbHg4 c4LeNDXp/q2rLlCFPlnhw+EFT79Rd1V0mutf3WKcIQcoNo/Or1rQUeuv7Wi8656SpbB+lqJIOzIzo zP4rDLgERILOJvbGflUs8SA/kji5nkcAuSkBQsczgmOHEgha3VRvEumnxd7QUSlqbnLaGO7j8hqUM 6nzQNVvC8ko7pgik8B3WlO/QIS2DkzBdVH/QJVSJmzcZ/IxhRu1yBOGEcAdL19tJnLtne+QIVhYdj kj8fddWBJwe9h59ROFXLXIFBgtngICLjMaAslGsbzW+wRH2J2MoP+aXsOgS+fF3r1ItkAv0uBr9ND 7On0zWNg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6TL4-006pvn-0U; Fri, 24 Nov 2023 10:23:10 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TK5-006pZT-38 for linux-arm-kernel@bombadil.infradead.org; Fri, 24 Nov 2023 10:22:10 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=3dnhOg/Z1S19RkFfJIGILiqGdJNmDmPL6z9OV6/m1cY=; b=RvA4JW6s1E7f5yQns6ic7ldj4t Ak7Es3dtZRKWMl6m90cWXnqFAheDuDsbnPkgAOcHIz1QSUgQdZ+n9P79qipjKG20CvRTyBYdHj1bX syWrkd8lwcPvBUWmykF/wNzmXtNXpYkdi8jMNTUBLUhjGRuRtwO3XA196Zz+eIsQYgB+pN6FfxU09 1O4J5DHYJjj6ZA5iQPp+McSitGFxkglbFomi1lj7iS0EoSnk00pIPVCmUyw5rJBIEVJUkwqy6UjdY kluVi3NDWP7YlQDoCbZ1pvWI01rVQQ1dGMsdPg4iyBH/UmcI4zGsFIhd3sY/85OOgp0rSLzTfEcXY dUIJR+zw==; Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by desiato.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6TJu-00DqXo-19 for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 10:22:07 +0000 Received: by mail-wm1-x34a.google.com with SMTP id 5b1f17b1804b1-40b3712ef28so7536715e9.2 for ; Fri, 24 Nov 2023 02:21:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700821298; x=1701426098; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3dnhOg/Z1S19RkFfJIGILiqGdJNmDmPL6z9OV6/m1cY=; b=D8gyXKmGt+mIuWf9M8qJzTp27CWIPHd6H9Z+whhTPzUHaduLosMa3e8sfL/QyrH085 THqrZO5iUW1b9N3Hmrk6q1VEV//L5XdGq77vbhf++EOHLFBTbDI92qgX9d+oq5TJrJzd 45zLQwPyRTLRzsVOeB+TZL8nLBvYQnxdKpi3rWZj3z8xIqf7eXM5Zb7UFGcZyLJVeaDG 3XssOm7loxGRRbnLgxcCH6oWz3DuJQhXYZnIFrQHPTRjOfyQAvQovFnzMOqA+bR27nl2 t24pgO+wijwKufgywkDzrXX6k/xKAh449Ra2KTyXJ3jNJsNvFAfgF5Jg7B61bscXFWc3 NQNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700821298; x=1701426098; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3dnhOg/Z1S19RkFfJIGILiqGdJNmDmPL6z9OV6/m1cY=; b=hr/ccmNdwodz2GpauaeO1vFNTfIN9juFsjRzol8roxJTpxoC/gPfQ1pNioyjFWYzBI +GDTVpbdRvqRPidl3SfYIu5iBsNYpc9sLxLmQeTyVfvlTliNPSPnoU8Mxs9bD1ZItj1o MzD+Nwnevz9S5m8GeGFmTWVSjnY++ZrxTv2k2dBBTAymfhBytjexlJ71vvg9xuLvmFCS DuTvsbz1eCpZHzThy4yDoemo1Gjb8wzcG8Vfv+IssqsBEdI4+sXeBPGFNTYldDA+elGc npyHg1CECgcCMqqde3P7UbeCsQIN4qgdEDYm/A5dOTti264FiI5Mux9JSzIYGU1kBnXU d35A== X-Gm-Message-State: AOJu0YxSUIwR9rwZf/K9hivt9jrJ15yRRy9eSXUw2QsO1bwZ6G7qUI57 vfPf0CBogujUxJvThT4YB9oWV3PLOqDMMUAICodq/biOVQAYKOicjDLNmpAdgE+HcFoBqrCDAWD 67Hz1Su0nj5PVZ1qQccQM3oIwcv9aHJcK6m+z6hKJsNVNHWhkrMF9PdILttyrFV1vqllDWi8REx g= X-Google-Smtp-Source: AGHT+IGW+yf1LlDo138/89zmHrXlZ0yUx8r22/TDbeaS8rT6c5bR1yQjgsK1nX4Iv+e8nRiirAyFWYh3 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:1e23:b0:40b:2c9e:e144 with SMTP id ay35-20020a05600c1e2300b0040b2c9ee144mr45185wmb.1.1700821297871; Fri, 24 Nov 2023 02:21:37 -0800 (PST) Date: Fri, 24 Nov 2023 11:19:17 +0100 In-Reply-To: <20231124101840.944737-41-ardb@google.com> Mime-Version: 1.0 References: <20231124101840.944737-41-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2126; i=ardb@kernel.org; h=from:subject; bh=fUAvuFuBmY0spPLjY96IzbCLIKESMxObqLpSTeEMtr4=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JITWhYtPvlesXrt5cNl/d+1XcDKmtDKpz/7Sa9CaF8bz8N O/yxZtZHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiy04w/JUVnn5xU8FrjztX 2Cb91VgUHKd2WNk/93f7pXRFg0rF+6EM/2s5rk3bPnf9mgtyua6q7+odLAtsxFbo7pNcVc6SIpO 4lgEA X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231124101840.944737-78-ardb@google.com> Subject: [PATCH v5 37/39] arm64: mm: avoid fixmap for early swapper_pg_dir updates From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_102158_646779_71230244 X-CRM114-Status: GOOD ( 13.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Early in the boot, when .rodata is still writable, we can poke swapper_pg_dir entries directly, and there is no need to go through the fixmap. After a future patch, we will enter the kernel with swapper_pg_dir already active, and early swapper_pg_dir updates for creating the fixmap page table hierarchy itself cannot go through the fixmap for obvious reaons. So let's keep track of whether rodata is writable, and update the descriptor directly in that case. As the same reasoning applies to early KASAN init, make the function noinstr as well. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/mmu.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index cb6b756dbb56..d36ba0811f5e 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -55,6 +55,8 @@ EXPORT_SYMBOL(kimage_voffset); u32 __boot_cpu_mode[] = { BOOT_CPU_MODE_EL2, BOOT_CPU_MODE_EL1 }; +static bool rodata_is_rw __ro_after_init = true; + /* * The booting CPU updates the failed status @__early_cpu_boot_status, * with MMU turned off. @@ -71,10 +73,21 @@ EXPORT_SYMBOL(empty_zero_page); static DEFINE_SPINLOCK(swapper_pgdir_lock); static DEFINE_MUTEX(fixmap_lock); -void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) +void noinstr set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) { pgd_t *fixmap_pgdp; + /* + * Don't bother with the fixmap if swapper_pg_dir is still mapped + * writable in the kernel mapping. + */ + if (rodata_is_rw) { + WRITE_ONCE(*pgdp, pgd); + dsb(ishst); + isb(); + return; + } + spin_lock(&swapper_pgdir_lock); fixmap_pgdp = pgd_set_fixmap(__pa_symbol(pgdp)); WRITE_ONCE(*fixmap_pgdp, pgd); @@ -628,6 +641,7 @@ void mark_rodata_ro(void) * to cover NOTES and EXCEPTION_TABLE. */ section_size = (unsigned long)__init_begin - (unsigned long)__start_rodata; + WRITE_ONCE(rodata_is_rw, false); update_mapping_prot(__pa_symbol(__start_rodata), (unsigned long)__start_rodata, section_size, PAGE_KERNEL_RO);