From patchwork Fri Nov 11 17:11:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040648 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6AF22C433FE for ; Fri, 11 Nov 2022 17:33:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=NvUyei26wyYKhECAN2LhbS5UFaZFlzMYzQJqPzLbh+k=; b=knHhL6g71y23Yu ztjHCKXJJw0xdJLGU//XfDYGr8GU+VZYtuiS2etCVGQc8+2aaJQMNJ8WrtXQWfn9uTioMkAdGCFIZ v2/CLoNjr4EZYV4vfAqe0A6yefFv9n1aAdA3uSUlG8+a6wpO6ZKAkOUsG+TECKQKjTWXR+wt5LWPw gjLjyCvr3r2Pxn5lcNJDvWIfVr4k6DJFNfBEyoTrM7xYQcSNaPmktjJNkKAglvKnpe7XPwrM2FKDV Bb9wXW286IOsKe0XrqTMxat7Zyb4hvg8/ff4sad3oVUie05upIDlafG9Em9LQtdRSo1ECLtMUBPDk jNBV0WLuxxCOkRKAFphA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXtF-00HI9H-Lm; Fri, 11 Nov 2022 17:32:30 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXam-00H84T-2r for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:25 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 87FCB6205C; Fri, 11 Nov 2022 17:13:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 14B68C43144; Fri, 11 Nov 2022 17:13:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186802; bh=nVh7PpTz2KdcicpZhASftr31fDniIS6jDeYMIEAwpMk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=N2rHcuwj/kGNwdBMPMFBkTD8haqs3ceZTnZ0icotU3NYixQ2KmAmV3rCjW1FAhfjo /OOrVugzbV8x7ZjbffvAFPRk9VlGRllASlTfqZiY/GWCJZ/aJn14Uce+zLArtX/lzb DROFCt0hEMWJxBwqTWuAXcO1kyJEyKWfCMztICDf6/M5IVAMA+xB4YrZNGpzuJGcvl AV94iV2rrdDPJ7lVaDAXM2tqcWIabGiOIYEj14SizQUYkSakZl20Zk7emJw1bEN4xA UN+JMqevYcysG3yW9JyWwgQB57CZfXD87/1JDppv4O9u5YWOsut8P1hLgLY5Vb1b7b xJ/FJYUpLMQ8g== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 28/33] arm64: mm: avoid fixmap for early swapper_pg_dir updates Date: Fri, 11 Nov 2022 18:11:56 +0100 Message-Id: <20221111171201.2088501-29-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2146; i=ardb@kernel.org; h=from:subject; bh=nVh7PpTz2KdcicpZhASftr31fDniIS6jDeYMIEAwpMk=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJXpLz7olDX386Tge6wUaBufeoF3qkK9fAcEK6G tBxD6OKJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CVwAKCRDDTyI5ktmPJNWIDA C0rGbikX4v7Gn/0O/6pGa94oPvMj99XINVo1AxyJWuYKQAJLPAf7EfBLZNJp6e0ztFpdodSTMx+GNa 30h0XeMrlb1DWmako6n/ta71dAOhwe8x6FSQq4jBLfBkO9gNqiVYNKznJyjEEo8ORa+5qQ+CBy01CI 5jzLAQBWZBM2HUHP9SWfKMPIfEzjw8HeS7gFA4Hmb0W7WrY7Z0N4hA90wfxgHBMho7L9vsubfUrQJR /76xjwbgv6YRe5p/5IR+9xFXoVDqEMkCDftnU1xbqKpygrAha8Ahf7iu8nCPl98UvJW5sezybN84zR jFp1cS/DvrqPcK+9qQ+kPF1+rjtwZn0w2NU43kszU8TcVxzIBKImeWRfvGdR4u7H9HfFDUD1Xvqz7c yRE7gMXuWUQmqO61biUx6c8n9+OJG4H/InzOLaTGkxBBlqoc4tFTGttynDPZRzaw8ECjJm3r5gPeur hcv01A9EMvtOynb0ak1POSNrLxxGJ+HcsBhXTBWWaTGns= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091324_275138_809C8C52 X-CRM114-Status: GOOD ( 15.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Early in the boot, when .rodata is still writable, we can poke swapper_pg_dir entries directly, and there is no need to go through the fixmap. After a future patch, we will enter the kernel with swapper_pg_dir already active, and early swapper_pg_dir updates for creating the fixmap page table hierarchy itself cannot go through the fixmap for obvious reaons. So let's keep track of whether rodata is writable, and update the descriptor directly in that case. As the same reasoning applies to early KASAN init, make the function noinstr as well. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/mmu.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 0c35e1f195678695..68e66b979fc3ac5d 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -58,6 +58,8 @@ EXPORT_SYMBOL(kimage_voffset); u32 __boot_cpu_mode[] = { BOOT_CPU_MODE_EL2, BOOT_CPU_MODE_EL1 }; +static bool rodata_is_rw __ro_after_init = true; + /* * The booting CPU updates the failed status @__early_cpu_boot_status, * with MMU turned off. @@ -78,10 +80,21 @@ static pud_t bm_pud[PTRS_PER_PUD] __page_aligned_bss __maybe_unused; static DEFINE_SPINLOCK(swapper_pgdir_lock); static DEFINE_MUTEX(fixmap_lock); -void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) +void noinstr set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) { pgd_t *fixmap_pgdp; + /* + * Don't bother with the fixmap if swapper_pg_dir is still mapped + * writable in the kernel mapping. + */ + if (rodata_is_rw) { + WRITE_ONCE(*pgdp, pgd); + dsb(ishst); + isb(); + return; + } + spin_lock(&swapper_pgdir_lock); fixmap_pgdp = pgd_set_fixmap(__pa_symbol(pgdp)); WRITE_ONCE(*fixmap_pgdp, pgd); @@ -615,6 +628,7 @@ void mark_rodata_ro(void) * to cover NOTES and EXCEPTION_TABLE. */ section_size = (unsigned long)__init_begin - (unsigned long)__start_rodata; + WRITE_ONCE(rodata_is_rw, false); update_mapping_prot(__pa_symbol(__start_rodata), (unsigned long)__start_rodata, section_size, PAGE_KERNEL_RO);