From patchwork Fri May 12 14:57:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 13239434 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CC54C77B7C for ; Fri, 12 May 2023 14:58:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 61BF76B0074; Fri, 12 May 2023 10:58:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5A4396B0075; Fri, 12 May 2023 10:58:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4456D6B0078; Fri, 12 May 2023 10:58:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3710F6B0074 for ; Fri, 12 May 2023 10:58:03 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 06875140F4E for ; Fri, 12 May 2023 14:58:02 +0000 (UTC) X-FDA: 80781908046.27.7E373F1 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf05.hostedemail.com (Postfix) with ESMTP id 19113100015 for ; Fri, 12 May 2023 14:58:00 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=UkPLrlyF; spf=pass (imf05.hostedemail.com: domain of bjorn@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bjorn@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1683903481; a=rsa-sha256; cv=none; b=zNC2pgyqp8AonTNW+wsGBtxMpd91WFZry01ClbPxobUnvFuW3PLKOUSFlHAtpXh1etofbS XVuxUKKhoAmVwsupmCBXIQAS7xRKdjy7TAB3YgPVbFqgdDL4DZ9WWi+mfyIdmq2XXYdJb3 q5+YFIPWO0q8ZwsSO4hoYP0fHmiEgwo= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=UkPLrlyF; spf=pass (imf05.hostedemail.com: domain of bjorn@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bjorn@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1683903481; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bXJFTabP/eR2shGXG2ZpQufidqoIkTMUZda4PVMUggs=; b=FNYGuFZF0WCctmLTEiUozXaORutvzTD+2BoNarR4VBckb9PMW9F03mGJdu7TIKLScDvZil +olI1NXo4V0ZbG0NQM/d24Wncyf6LLFSQChWMwMGB9VZkLsChnT/krV5pUMB8o4HUwns8t txOgZclXw3n3xCi1gsyqd3Uf2TKUNiE= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 10D946574C; Fri, 12 May 2023 14:58:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9CEA8C4339B; Fri, 12 May 2023 14:57:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683903479; bh=nJDJ9NE+3mcyJNnGkZl+cobDBvPDump2PA8ld/F5o/w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UkPLrlyFezs3jORrA3aROLm9iXC6Kr9vZjNIJzD6TiCZIqDJsxH49o0Pu+gEOK/gU VQeKq8IgWZHG4gxyuGcDrQwf0pp5Ti0KFT8is/rChRqO0i3zkfDEZmL9fQi/Yrx6Wj gHPD02kEB1754XbX7Dra6eV9gfEWk+l0+gyV+8AtChmvojBDWyWK4jNMnQ6wv8Ytdv W7YJLyXvLt9mx73HO6vwf+V0/FvXaM4esp+YR7xgUlVMxUGXnkUQ0kiaiaLLtHbtN/ QB9T/Gofz00zxK9cWCo+6XUlba0SMqDf2WOJ/jm6328ZydxnvV7MKyLnlDsJQZIFjQ M+xECs9iopGtA== From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-riscv@lists.infradead.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-kernel@vger.kernel.org, linux-mm@kvack.org, David Hildenbrand , Oscar Salvador , virtualization@lists.linux-foundation.org, linux@rivosinc.com, Alexandre Ghiti Subject: [PATCH 1/7] riscv: mm: Pre-allocate PGD leaves to avoid synchronization Date: Fri, 12 May 2023 16:57:31 +0200 Message-Id: <20230512145737.985671-2-bjorn@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230512145737.985671-1-bjorn@kernel.org> References: <20230512145737.985671-1-bjorn@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 19113100015 X-Stat-Signature: b7sriyw59dkm53cboyf9y8izw4ehdt9x X-HE-Tag: 1683903480-600913 X-HE-Meta: U2FsdGVkX1/Z/2HZ1SC6oM5FRswpfcM1EwbNRiy7gqsgf6XlHAdoNyemJtxq2Vxg59Su3nysfbWy1Zd+Fr1+8x+xudnbtGlIcIXB4/ZglbUaRp5uzHnGM7bjkEdIyNwEmuKTE5qBEjP+6La9GydQlM7WDdULrUaz1s4DEpBjhQAnC5/9in6jTga0j5pXRmBC0M0Db0KMRg44g9gMstpM50bmwcToCewKSjJ9LoZrpcWfKq3KE9bJM2pw4py9qdNCA6TJfQcbKCp1S6zA3OAAmsnYnH7d3YSufYGLT+MizU36bbXtqUtDw88Ea8i1hheybSSHac0vE/ue1VlQSkjGDmDmW1RAi3XfmG2D/aPFYhs5MRTV0Hy5Eox6jkH3gN1oqGyBSCjJM7RE4YWNIQoMuyLfsUQUunKKRFvQJzd+XJMt+v3BDQEkv5Jcas0NsicggarEj1RjCV75wBg6cxewKWSp4Zc0LyC75Xv7jGNzKihr961vXpd+BPg2lkiA9Zod0dWDD2LbiiKHULb2J5dlNKhFjOZMG3Ts2XqE3IibibISxIdDy11vsAQKytfi3dBt4I+UPoNaLKmTpvATRkmYLIA+slQQSu563U0mc2MkyRk98lqLcIU4JA2YvS4UEm/zYlQR4a664HQaebxajMhVZuN0f90hdbkxb2LdZumhatov9A3L4U06f52XUTvLbn/uypMKrm4RaUNValME01vy61SKObfyjHWxn+LNP/s71sAqOKsaGLJaUL7MlnDRPt0SYinagIV5piFAYNFOAz1eTaTWHsGg3IfKOEYG1vn+2Y0PsES/rLzxKIDvEivLTrrqmmU0aFWk1O/XKKM3Yv6L5bHM91FtcUPP77rRl8csxt22NOUYQF+KPeSU+jPwKMjT1FkhsMfPvCe4RQ5WiI1Bas0/E0PTmbMFjcnxgJ82QpFmIvGUhboBIzrCb5ZyA6VjWTYZXpg5TeNxT/SR75B dudAlWew nZ7RW7gwIsZquosPh68Xf6NkBhHviyqhGp9kXAbDBiiCmdWp1XVLav6Fa2bPdgyveniDuTDQQkEw+LR4A/1nqXp9SLzn5CvQqHVrmgIZ/3qYIZOnIJqV0MUrXS7ex1Dkl3VpBdspxc7UXrGWqyKNOqfCBFmrwFWW7lZostxYmtSoKAxURLDSdtSMo5rU0rt5dYM3AkD+Mea/JvV0zhXBTdVAygt7TFkAYA22Of5eoRmm4ZhIgGh/cM9dEumN+0QyzJsnR3QprcijWvm/pp90vicztAP0M1hNftVwufDX59549XOBas7I2Q+yfozx7F2ur3wJf7s650wc9QTWLLdXz/SoCeR115w5aEum8RG90vAFluH0YsJC+TbyuaLXk/+G7N0Zydlmo8xRquKkTZ2IKhpM/Tlt/6KybNu8p X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Björn Töpel The RISC-V port copies PGD from init_mm to all userland pages-tables, which means that when the PGD level of the init_mm table is changed, other page-tables has to be updated. One way to avoid synchronizing page-tables is to pre-allocate the pages that are copied (need to be synchronized). For memory hotswapping builds, prefer to waste some pages, rather than do explicit synchronization. Prepare the RISC-V port for memory add/remove, by getting rid of PGD synchronization. Pre-allocate vmemmap, and direct map pages. This will roughly waste ~128 worth of 4K pages. Note that this is only done for memory hotswap enabled configuration. Signed-off-by: Björn Töpel --- arch/riscv/include/asm/kasan.h | 4 +- arch/riscv/mm/init.c | 86 ++++++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+), 2 deletions(-) diff --git a/arch/riscv/include/asm/kasan.h b/arch/riscv/include/asm/kasan.h index 0b85e363e778..e6a0071bdb56 100644 --- a/arch/riscv/include/asm/kasan.h +++ b/arch/riscv/include/asm/kasan.h @@ -6,8 +6,6 @@ #ifndef __ASSEMBLY__ -#ifdef CONFIG_KASAN - /* * The following comment was copied from arm64: * KASAN_SHADOW_START: beginning of the kernel virtual addresses. @@ -34,6 +32,8 @@ */ #define KASAN_SHADOW_START ((KASAN_SHADOW_END - KASAN_SHADOW_SIZE) & PGDIR_MASK) #define KASAN_SHADOW_END MODULES_LOWEST_VADDR + +#ifdef CONFIG_KASAN #define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL) void kasan_init(void); diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 747e5b1ef02d..d2595cc33a1c 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -31,6 +31,7 @@ #include #include #include +#include #include "../kernel/head.h" @@ -156,6 +157,90 @@ static void __init print_vm_layout(void) static void print_vm_layout(void) { } #endif /* CONFIG_DEBUG_VM */ +#ifdef CONFIG_MEMORY_HOTPLUG +/* + * Pre-allocates page-table pages for a specific area in the kernel + * page-table. Only the level which needs to be synchronized between + * all page-tables is allocated because the synchronization can be + * expensive. + */ +static void __init preallocate_pgd_pages_range(unsigned long start, unsigned long end, + const char *area) +{ + unsigned long addr; + const char *lvl; + + for (addr = start; addr < end; addr = ALIGN(addr + 1, PGDIR_SIZE)) { + pgd_t *pgd = pgd_offset_k(addr); + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + + lvl = "p4d"; + p4d = p4d_alloc(&init_mm, pgd, addr); + if (!p4d) + goto failed; + + if (pgtable_l5_enabled) + continue; + + /* + * The goal here is to allocate all possibly required + * hardware page tables pointed to by the top hardware + * level. + * + * On 4-level systems, the P4D layer is folded away + * and the above code does no preallocation. Below, + * go down to the pud _software_ level to ensure the + * second hardware level is allocated on 4-level + * systems too. + */ + lvl = "pud"; + pud = pud_alloc(&init_mm, p4d, addr); + if (!pud) + goto failed; + + if (pgtable_l4_enabled) + continue; + /* + * The goal here is to allocate all possibly required + * hardware page tables pointed to by the top hardware + * level. + * + * On 3-level systems, the PUD layer is folded away + * and the above code does no preallocation. Below, + * go down to the pmd _software_ level to ensure the + * second hardware level is allocated on 3-level + * systems too. + */ + lvl = "pmd"; + pmd = pmd_alloc(&init_mm, pud, addr); + if (!pmd) + goto failed; + } + + return; + +failed: + + /* + * The pages have to be there now or they will be missing in + * process page-tables later. + */ + panic("Failed to pre-allocate %s pages for %s area\n", lvl, area); +} + +#define PAGE_END KASAN_SHADOW_START +#endif + +static void __init prepare_memory_hotplug(void) +{ +#ifdef CONFIG_MEMORY_HOTPLUG + preallocate_pgd_pages_range(VMEMMAP_START, VMEMMAP_END, "vmemmap"); + preallocate_pgd_pages_range(PAGE_OFFSET, PAGE_END, "direct map"); +#endif +} + void __init mem_init(void) { #ifdef CONFIG_FLATMEM @@ -164,6 +249,7 @@ void __init mem_init(void) swiotlb_init(max_pfn > PFN_DOWN(dma32_phys_limit), SWIOTLB_VERBOSE); memblock_free_all(); + prepare_memory_hotplug(); print_vm_layout(); }