From patchwork Tue Sep 12 14:16:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C80F2CA0EEC for ; Tue, 12 Sep 2023 14:20:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=EsYhTljosz0+LCNuhjqzFoPMXIJNI+ZjuqZzaDBNbUM=; b=1sGqn7kzaAGvuaWgS4klHGJCB+ J5HD4SgUJgKmdPneL3v5Buk26qIkHE5BGLTXOKP0PS6XJFJ5aTzfOhMirhQj9DujrPz1bBuwf/qHN 4PST9DO03xX5cHktLczG2c2a1wCVbEDM7tM7UHCNvSqowtqJdALAFJgqHWXu7VmExvBy5FiR2TeRv lD/G2b+FnaVckP1RVzexQ9la/y8IGSS24ethr9AQIgvnAq6p3Qu1xKKh5WE6bMxkpnyMG64d6rSB7 Sb7GcqltVzK/ZEuYA2z00qhVhOVVy70pBYMDVg7O9tuAHgZ9oQhL91OX0ko2GjdqFw+j2KXBslYnu Dv0AGmFw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4F8-003Xqn-1e; Tue, 12 Sep 2023 14:19:54 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4E4-003WvP-2Z for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:03 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-30932d15a30so4056232f8f.1 for ; Tue, 12 Sep 2023 07:18:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528326; x=1695133126; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lu+998G42yoYzQEKfrWeyih/jY59r+JYoyBx68HGSFI=; b=gPxOYY+fN0UwnB4wb/iop5QxZ4AnM0BIt21P1FocunIgatNI0e/gcCMa3lFEvz+6vG JyHERBKwTwJsCMfs1ylDkRVOQ6sbh6IVaCPZ8puDVs4kpLqWlJVNfSVQil2IlH4EejTK ydLrsyC2vi9xiTGeloi9AGG+uS44xo9RG/xbLupGxCkryNy/L9L+GPx+ftHOYPVGgFaJ NmPC874EUbRydqSp3kTcXes2G2wuJIMHf8zw4PvuWqrVyjTQQrNLGYounKO3PxU946wE bUbTYpN4CzQWVxh5wvmPxYDbRjgbHfylAvmVd6mACiw+o+nxx8GU+yLMn36YLFDDnjhI 1/Lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528326; x=1695133126; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lu+998G42yoYzQEKfrWeyih/jY59r+JYoyBx68HGSFI=; b=b+5rvWrPtmHWtFtTx8qpYPr27TWNv4figLqnW1hmUDSlFAgBNL1D+N2y027TSmiGmF gxrzyXhJVptIqH5J0EIqthAz/JWmySy8tuJIv/jJ6T+wWkm74ti9qxXc3UOCgRDdnJuz mDbnGUAdxCxQrQJhzlZQex3Wn3a7LaTCTH8epyhf0YHty8B6QrBwd+uIIYT+Q44mht85 yQjwHPCaI7q8Rve2Mk8tRWM2O8x7hIU4+ulBaRP5hGl8J9SFgklzCGJHbfk4KGJy4SNk fGn6XAMXuBSzYXQQ4bIzaIpVxfDPnlJ1bcXQWRGeHEIX21yWbM7QQskzMySXvIBCPAyv cEnA== X-Gm-Message-State: AOJu0Yy+VOi2BoS67g+PRNSu4MM1RWnORs37wFTctMY9aiP3wuoWQKhx ACk2nJeJ6x7KZ2sRuqv+wuAaGMkD3BVVaKDGtsuc+336XH1oEqj3eVtyb/uEfZYEPBLmTpb6KHv nJLHCPUYn4uvi0GRDEzXxyp5Zpn0mA6uZoHC6MHZYv1VEg/688VctiWconhx5nySRxtUD29slM5 s= X-Google-Smtp-Source: AGHT+IHa/pWhDddtcvlLtch8bevNYQuYRuskg5kcTws4jb1bcdKCg8dFqCBDt4bemPwUfCWsdjfqwqTU X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:d1e4:0:b0:31f:9838:dfbc with SMTP id g4-20020adfd1e4000000b0031f9838dfbcmr107432wrd.11.1694528325803; Tue, 12 Sep 2023 07:18:45 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:27 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2122; i=ardb@kernel.org; h=from:subject; bh=o9DIE+pNddG54HtQ0IrOHmtMLIY+G8vnueC1wug+zzo=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWhaM/EB6s4TpzfVaOd5bvk7vpXmeKvHl/eqLOhJLeLe 0OxEK93RykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjIyX+MDO+unTpU8napMktV a3/+u3IpyU9JEt+9HeecywqM6Lpy8QnDf2/lmJnftKomHJuwcZXr4e1OR8w/frym5DLnvCXndLu 6fH4A X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-100-ardb@google.com> Subject: [PATCH v4 37/61] arm64: mm: avoid fixmap for early swapper_pg_dir updates From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071848_891167_59D0D2B4 X-CRM114-Status: GOOD ( 15.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Early in the boot, when .rodata is still writable, we can poke swapper_pg_dir entries directly, and there is no need to go through the fixmap. After a future patch, we will enter the kernel with swapper_pg_dir already active, and early swapper_pg_dir updates for creating the fixmap page table hierarchy itself cannot go through the fixmap for obvious reaons. So let's keep track of whether rodata is writable, and update the descriptor directly in that case. As the same reasoning applies to early KASAN init, make the function noinstr as well. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/mmu.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 9d965c4df65d..ae2134be6be5 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -55,6 +55,8 @@ EXPORT_SYMBOL(kimage_voffset); u32 __boot_cpu_mode[] = { BOOT_CPU_MODE_EL2, BOOT_CPU_MODE_EL1 }; +static bool rodata_is_rw __ro_after_init = true; + /* * The booting CPU updates the failed status @__early_cpu_boot_status, * with MMU turned off. @@ -71,10 +73,21 @@ EXPORT_SYMBOL(empty_zero_page); static DEFINE_SPINLOCK(swapper_pgdir_lock); static DEFINE_MUTEX(fixmap_lock); -void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) +void noinstr set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) { pgd_t *fixmap_pgdp; + /* + * Don't bother with the fixmap if swapper_pg_dir is still mapped + * writable in the kernel mapping. + */ + if (rodata_is_rw) { + WRITE_ONCE(*pgdp, pgd); + dsb(ishst); + isb(); + return; + } + spin_lock(&swapper_pgdir_lock); fixmap_pgdp = pgd_set_fixmap(__pa_symbol(pgdp)); WRITE_ONCE(*fixmap_pgdp, pgd); @@ -628,6 +641,7 @@ void mark_rodata_ro(void) * to cover NOTES and EXCEPTION_TABLE. */ section_size = (unsigned long)__init_begin - (unsigned long)__start_rodata; + WRITE_ONCE(rodata_is_rw, false); update_mapping_prot(__pa_symbol(__start_rodata), (unsigned long)__start_rodata, section_size, PAGE_KERNEL_RO);