From patchwork Thu Apr 4 14:33:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13617955 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2490FCD1284 for ; Thu, 4 Apr 2024 14:33:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=eBaTh3+8IlBT0gcGQaalFQQFPP2jqN14gMZi+imd6Kk=; b=1uULsnqWPQsYb+ E2osXkgSCMDE/vVfUcbTN/UYM0v1ZMgbkJNpIRlUUo/Rmjov+i37eTr1z+kekcXMCpDnvQXQwa+ZA M5JroSKVrWHyyaXGcJ6NEsQbqxK0KKGR5DBzF8vbxnNenFRc5k63zxhWj3bhX0crfLJlapHV2fVsu fulLjL8S2V2d8oJI+x5wWCj/zjdgtq5zjZty7iV0qftZPeAVk0yca1PW/iefO+3S4mJWqx2ZniFfB 7vFnljDUQKNoxGHG++WS6gIX/8ovlB6Bf8tKXSMYig64bSP31yl4t+Kx5T0Z5Q+gSrvr4HhAY3Y6x C9tiVwBHTil19OCOVOAw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rsO9c-000000032zJ-3enn; Thu, 04 Apr 2024 14:33:24 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rsO9Z-000000032y1-2s5B for linux-arm-kernel@lists.infradead.org; Thu, 04 Apr 2024 14:33:23 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6F02DFEC; Thu, 4 Apr 2024 07:33:49 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6499A3F64C; Thu, 4 Apr 2024 07:33:17 -0700 (PDT) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Mark Rutland , Ard Biesheuvel , David Hildenbrand , Donald Dutile , Eric Chanudet Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 0/4] Speed up boot with faster linear map creation Date: Thu, 4 Apr 2024 15:33:04 +0100 Message-Id: <20240404143308.2224141-1-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240404_073321_805214_1E9062E9 X-CRM114-Status: GOOD ( 10.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi All, It turns out that creating the linear map can take a significant proportion of the total boot time, especially when rodata=full. And most of the time is spent waiting on superfluous tlb invalidation and memory barriers. This series reworks the kernel pgtable generation code to significantly reduce the number of those TLBIs, ISBs and DSBs. See each patch for details. The below shows the execution time of map_mem() across a couple of different systems with different RAM configurations. We measure after applying each patch and show the improvement relative to base (v6.9-rc2): | Apple M2 VM | Ampere Altra| Ampere Altra| Ampere Altra | VM, 16G | VM, 64G | VM, 256G | Metal, 512G ---------------|-------------|-------------|-------------|------------- | ms (%) | ms (%) | ms (%) | ms (%) ---------------|-------------|-------------|-------------|------------- base | 153 (0%) | 2227 (0%) | 8798 (0%) | 17442 (0%) no-cont-remap | 77 (-49%) | 431 (-81%) | 1727 (-80%) | 3796 (-78%) batch-barriers | 13 (-92%) | 162 (-93%) | 655 (-93%) | 1656 (-91%) no-alloc-remap | 11 (-93%) | 109 (-95%) | 449 (-95%) | 1257 (-93%) lazy-unmap | 6 (-96%) | 61 (-97%) | 257 (-97%) | 838 (-95%) This series applies on top of v6.9-rc2. All mm selftests pass. I've compile and boot tested various PAGE_SIZE and VA size configs. --- Changes since v1 [1] ==================== - Added Tested-by tags (thanks to Eric and Itaru) - Renamed ___set_pte() -> __set_pte_nosync() (per Ard) - Reordered patches (biggest impact & least controversial first) - Reordered alloc/map/unmap functions in mmu.c to aid reader - pte_clear() -> __pte_clear() in clear_fixmap_nosync() - Reverted generic p4d_index() which caused x86 build error. Replaced with unconditional p4d_index() define under arm64. [1] https://lore.kernel.org/linux-arm-kernel/20240326101448.3453626-1-ryan.roberts@arm.com/ Thanks, Ryan Ryan Roberts (4): arm64: mm: Don't remap pgtables per-cont(pte|pmd) block arm64: mm: Batch dsb and isb when populating pgtables arm64: mm: Don't remap pgtables for allocate vs populate arm64: mm: Lazily clear pte table mappings from fixmap arch/arm64/include/asm/fixmap.h | 5 +- arch/arm64/include/asm/mmu.h | 8 + arch/arm64/include/asm/pgtable.h | 13 +- arch/arm64/kernel/cpufeature.c | 10 +- arch/arm64/mm/fixmap.c | 11 + arch/arm64/mm/mmu.c | 377 +++++++++++++++++++++++-------- 6 files changed, 319 insertions(+), 105 deletions(-) -- 2.25.1