From patchwork Fri May 12 14:57:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 13239443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B2C64C77B7F for ; Fri, 12 May 2023 14:58:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=R5ptucGYfXgShQLysUNtOyAXg6jgNA4i+MVk+YucuCY=; b=rAUjR3CQebquoY A/USm6NmgCG2oSe7J0VyU9C3GQikExM1D5maQ5WNmGDtOeVro4qnJBYSjg4c+v4QhNLNNjXuN/CPP 1m+ewg25oYKSmXDx+y8VkLYam9jnXhNoxGX6mf/+FDRH8J8L9oNbOMeMt6EhWLnQs8cKxVIeYXfpD PMf1v7yPKTUkbgdCOgsCJHMkj5ZnhGU2GG1lPtbVhH4ivK+ye4M21KcMFMerTW3z4ZOHZS38MKaA+ ss8Nh3jwNNSzOZtL+filBKle1I4kCdR2VeOqzLMAW0RaTJ4FuHH6jZtJPN7nBMTNzoyitGIKZm9m0 pFFCxpH25tvyCCzwXBSw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pxUDb-00CFuB-18; Fri, 12 May 2023 14:58:03 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pxUDY-00CFtL-1Q for linux-riscv@lists.infradead.org; Fri, 12 May 2023 14:58:01 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 10D946574C; Fri, 12 May 2023 14:58:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9CEA8C4339B; Fri, 12 May 2023 14:57:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683903479; bh=nJDJ9NE+3mcyJNnGkZl+cobDBvPDump2PA8ld/F5o/w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UkPLrlyFezs3jORrA3aROLm9iXC6Kr9vZjNIJzD6TiCZIqDJsxH49o0Pu+gEOK/gU VQeKq8IgWZHG4gxyuGcDrQwf0pp5Ti0KFT8is/rChRqO0i3zkfDEZmL9fQi/Yrx6Wj gHPD02kEB1754XbX7Dra6eV9gfEWk+l0+gyV+8AtChmvojBDWyWK4jNMnQ6wv8Ytdv W7YJLyXvLt9mx73HO6vwf+V0/FvXaM4esp+YR7xgUlVMxUGXnkUQ0kiaiaLLtHbtN/ QB9T/Gofz00zxK9cWCo+6XUlba0SMqDf2WOJ/jm6328ZydxnvV7MKyLnlDsJQZIFjQ M+xECs9iopGtA== From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-riscv@lists.infradead.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-kernel@vger.kernel.org, linux-mm@kvack.org, David Hildenbrand , Oscar Salvador , virtualization@lists.linux-foundation.org, linux@rivosinc.com, Alexandre Ghiti Subject: [PATCH 1/7] riscv: mm: Pre-allocate PGD leaves to avoid synchronization Date: Fri, 12 May 2023 16:57:31 +0200 Message-Id: <20230512145737.985671-2-bjorn@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230512145737.985671-1-bjorn@kernel.org> References: <20230512145737.985671-1-bjorn@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230512_075800_555374_657CBC25 X-CRM114-Status: GOOD ( 21.23 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Björn Töpel The RISC-V port copies PGD from init_mm to all userland pages-tables, which means that when the PGD level of the init_mm table is changed, other page-tables has to be updated. One way to avoid synchronizing page-tables is to pre-allocate the pages that are copied (need to be synchronized). For memory hotswapping builds, prefer to waste some pages, rather than do explicit synchronization. Prepare the RISC-V port for memory add/remove, by getting rid of PGD synchronization. Pre-allocate vmemmap, and direct map pages. This will roughly waste ~128 worth of 4K pages. Note that this is only done for memory hotswap enabled configuration. Signed-off-by: Björn Töpel --- arch/riscv/include/asm/kasan.h | 4 +- arch/riscv/mm/init.c | 86 ++++++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+), 2 deletions(-) diff --git a/arch/riscv/include/asm/kasan.h b/arch/riscv/include/asm/kasan.h index 0b85e363e778..e6a0071bdb56 100644 --- a/arch/riscv/include/asm/kasan.h +++ b/arch/riscv/include/asm/kasan.h @@ -6,8 +6,6 @@ #ifndef __ASSEMBLY__ -#ifdef CONFIG_KASAN - /* * The following comment was copied from arm64: * KASAN_SHADOW_START: beginning of the kernel virtual addresses. @@ -34,6 +32,8 @@ */ #define KASAN_SHADOW_START ((KASAN_SHADOW_END - KASAN_SHADOW_SIZE) & PGDIR_MASK) #define KASAN_SHADOW_END MODULES_LOWEST_VADDR + +#ifdef CONFIG_KASAN #define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL) void kasan_init(void); diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 747e5b1ef02d..d2595cc33a1c 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -31,6 +31,7 @@ #include #include #include +#include #include "../kernel/head.h" @@ -156,6 +157,90 @@ static void __init print_vm_layout(void) static void print_vm_layout(void) { } #endif /* CONFIG_DEBUG_VM */ +#ifdef CONFIG_MEMORY_HOTPLUG +/* + * Pre-allocates page-table pages for a specific area in the kernel + * page-table. Only the level which needs to be synchronized between + * all page-tables is allocated because the synchronization can be + * expensive. + */ +static void __init preallocate_pgd_pages_range(unsigned long start, unsigned long end, + const char *area) +{ + unsigned long addr; + const char *lvl; + + for (addr = start; addr < end; addr = ALIGN(addr + 1, PGDIR_SIZE)) { + pgd_t *pgd = pgd_offset_k(addr); + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + + lvl = "p4d"; + p4d = p4d_alloc(&init_mm, pgd, addr); + if (!p4d) + goto failed; + + if (pgtable_l5_enabled) + continue; + + /* + * The goal here is to allocate all possibly required + * hardware page tables pointed to by the top hardware + * level. + * + * On 4-level systems, the P4D layer is folded away + * and the above code does no preallocation. Below, + * go down to the pud _software_ level to ensure the + * second hardware level is allocated on 4-level + * systems too. + */ + lvl = "pud"; + pud = pud_alloc(&init_mm, p4d, addr); + if (!pud) + goto failed; + + if (pgtable_l4_enabled) + continue; + /* + * The goal here is to allocate all possibly required + * hardware page tables pointed to by the top hardware + * level. + * + * On 3-level systems, the PUD layer is folded away + * and the above code does no preallocation. Below, + * go down to the pmd _software_ level to ensure the + * second hardware level is allocated on 3-level + * systems too. + */ + lvl = "pmd"; + pmd = pmd_alloc(&init_mm, pud, addr); + if (!pmd) + goto failed; + } + + return; + +failed: + + /* + * The pages have to be there now or they will be missing in + * process page-tables later. + */ + panic("Failed to pre-allocate %s pages for %s area\n", lvl, area); +} + +#define PAGE_END KASAN_SHADOW_START +#endif + +static void __init prepare_memory_hotplug(void) +{ +#ifdef CONFIG_MEMORY_HOTPLUG + preallocate_pgd_pages_range(VMEMMAP_START, VMEMMAP_END, "vmemmap"); + preallocate_pgd_pages_range(PAGE_OFFSET, PAGE_END, "direct map"); +#endif +} + void __init mem_init(void) { #ifdef CONFIG_FLATMEM @@ -164,6 +249,7 @@ void __init mem_init(void) swiotlb_init(max_pfn > PFN_DOWN(dma32_phys_limit), SWIOTLB_VERBOSE); memblock_free_all(); + prepare_memory_hotplug(); print_vm_layout(); } From patchwork Fri May 12 14:57:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 13239442 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 39E3EC77B7C for ; Fri, 12 May 2023 14:58:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=p8QHxHaGCAtbTw7RxEOkU/3PgClAK5sSJxgHIgv/PI0=; b=lQrrgW0yJuwHM/ aKMp4Abfj1a3xflJSaHhlPxxtPcRX97xHLes801mRYBelxaCfFNdVCZM7c16tAYQbzwuN2GJT97/C XaCO7Ordu0/usNsEf68b7dBy4et4AZ0RyPbFty7qO5bZP3bPSYeCJhSAbP4+zU+LYPCGp9CYq/zPV itT2uXoD74phZqRI7++4jPLH7OkdZ72LP0jIwH5Nh1P71uEgrneKfa4hlyd0/AmM+4GCtp5j9E5kc GlDyxBSgKemN9l+zB97cpuV0komGdwiHWFf5kSTtK+v9OvxJmjSZ/1fxw5RGl7F4cTKjUkfTaFwBe 2bCdfn4Dbyx5NetKomtg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pxUDe-00CFw3-38; Fri, 12 May 2023 14:58:06 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pxUDb-00CFuI-2s for linux-riscv@lists.infradead.org; Fri, 12 May 2023 14:58:05 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 54A0D60F02; Fri, 12 May 2023 14:58:03 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 057B2C433EF; Fri, 12 May 2023 14:57:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683903482; bh=TbHgG2w0XhFAPXkR+9W7hs3AnwbEfk/0KY6kqBQIJeI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rWaW1yO4zoG2gWhzY8P7BBZ0myq/alPhT7G9vRjXjVAcXidr8fdCS6X5CaHFGusk7 qjVS7b8XygHKs5JocjZwRQDxE+2oIuGJdZK09eCTMn0jLx8sOjkhFdpD3WL7S+BBgX r4z2SQVi2ZV24FFoPCpEtzWv+8VokXA3/LhdoSwVI7mcsZQJ/GkKl2yHLw+2PwlzvI 8GE3RPwss0CYTl9KkCRlDemA4GHpeMZ15fFn4z7MXRJg1TGizz45LBk1tMwq4mOHkO 9ILUr0Iz6q6JnHnIL/zxOMu9dQ/j4NYpc9GZAgm4PtOIR5ohWNiccDnmE/1e2KtNht 3fAZL7Z6pcaJA== From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-riscv@lists.infradead.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-kernel@vger.kernel.org, linux-mm@kvack.org, David Hildenbrand , Oscar Salvador , virtualization@lists.linux-foundation.org, linux@rivosinc.com, Alexandre Ghiti Subject: [PATCH 2/7] riscv: mm: Change attribute from __init to __meminit for page functions Date: Fri, 12 May 2023 16:57:32 +0200 Message-Id: <20230512145737.985671-3-bjorn@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230512145737.985671-1-bjorn@kernel.org> References: <20230512145737.985671-1-bjorn@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230512_075804_009350_FD0261DF X-CRM114-Status: GOOD ( 14.69 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Björn Töpel Prepare for memory hot add/remove support by changing from __init to __meminit for the page-table functions that are used by the upcoming arch specific callbacks. Changing the __init attribute to __meminit, avoids that the functions are removed after init. __meminit keeps the functions after init, if memory hotplugging is enabled for the build. Signed-off-by: Björn Töpel --- arch/riscv/include/asm/mmu.h | 2 +- arch/riscv/include/asm/pgtable.h | 2 +- arch/riscv/mm/init.c | 49 ++++++++++++++------------------ 3 files changed, 24 insertions(+), 29 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index 0099dc116168..9e5d4f37ba2e 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -22,7 +22,7 @@ typedef struct { #endif } mm_context_t; -void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, +void __meminit create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); #endif /* __ASSEMBLY__ */ diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 2258b27173b0..a4cdcb689959 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -147,7 +147,7 @@ struct pt_alloc_ops { #endif }; -extern struct pt_alloc_ops pt_ops __initdata; +extern struct pt_alloc_ops pt_ops __meminitdata; #ifdef CONFIG_MMU /* Number of PGD entries that a user-mode program can use */ diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index d2595cc33a1c..e974ff6ef036 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -356,7 +356,7 @@ static void __init setup_bootmem(void) } #ifdef CONFIG_MMU -struct pt_alloc_ops pt_ops __initdata; +struct pt_alloc_ops pt_ops __meminitdata; pgd_t swapper_pg_dir[PTRS_PER_PGD] __page_aligned_bss; pgd_t trampoline_pg_dir[PTRS_PER_PGD] __page_aligned_bss; @@ -418,7 +418,7 @@ static inline pte_t *__init get_pte_virt_fixmap(phys_addr_t pa) return (pte_t *)set_fixmap_offset(FIX_PTE, pa); } -static inline pte_t *__init get_pte_virt_late(phys_addr_t pa) +static inline pte_t *__meminit get_pte_virt_late(phys_addr_t pa) { return (pte_t *) __va(pa); } @@ -437,7 +437,7 @@ static inline phys_addr_t __init alloc_pte_fixmap(uintptr_t va) return memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE); } -static phys_addr_t __init alloc_pte_late(uintptr_t va) +static phys_addr_t __meminit alloc_pte_late(uintptr_t va) { unsigned long vaddr; @@ -447,9 +447,8 @@ static phys_addr_t __init alloc_pte_late(uintptr_t va) return __pa(vaddr); } -static void __init create_pte_mapping(pte_t *ptep, - uintptr_t va, phys_addr_t pa, - phys_addr_t sz, pgprot_t prot) +static void __meminit create_pte_mapping(pte_t *ptep, uintptr_t va, phys_addr_t pa, phys_addr_t sz, + pgprot_t prot) { uintptr_t pte_idx = pte_index(va); @@ -503,7 +502,7 @@ static pmd_t *__init get_pmd_virt_fixmap(phys_addr_t pa) return (pmd_t *)set_fixmap_offset(FIX_PMD, pa); } -static pmd_t *__init get_pmd_virt_late(phys_addr_t pa) +static pmd_t *__meminit get_pmd_virt_late(phys_addr_t pa) { return (pmd_t *) __va(pa); } @@ -520,7 +519,7 @@ static phys_addr_t __init alloc_pmd_fixmap(uintptr_t va) return memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE); } -static phys_addr_t __init alloc_pmd_late(uintptr_t va) +static phys_addr_t __meminit alloc_pmd_late(uintptr_t va) { unsigned long vaddr; @@ -530,9 +529,8 @@ static phys_addr_t __init alloc_pmd_late(uintptr_t va) return __pa(vaddr); } -static void __init create_pmd_mapping(pmd_t *pmdp, - uintptr_t va, phys_addr_t pa, - phys_addr_t sz, pgprot_t prot) +static void __meminit create_pmd_mapping(pmd_t *pmdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, + pgprot_t prot) { pte_t *ptep; phys_addr_t pte_phys; @@ -568,7 +566,7 @@ static pud_t *__init get_pud_virt_fixmap(phys_addr_t pa) return (pud_t *)set_fixmap_offset(FIX_PUD, pa); } -static pud_t *__init get_pud_virt_late(phys_addr_t pa) +static pud_t *__meminit get_pud_virt_late(phys_addr_t pa) { return (pud_t *)__va(pa); } @@ -586,7 +584,7 @@ static phys_addr_t __init alloc_pud_fixmap(uintptr_t va) return memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE); } -static phys_addr_t alloc_pud_late(uintptr_t va) +static phys_addr_t __meminit alloc_pud_late(uintptr_t va) { unsigned long vaddr; @@ -606,7 +604,7 @@ static p4d_t *__init get_p4d_virt_fixmap(phys_addr_t pa) return (p4d_t *)set_fixmap_offset(FIX_P4D, pa); } -static p4d_t *__init get_p4d_virt_late(phys_addr_t pa) +static p4d_t *__meminit get_p4d_virt_late(phys_addr_t pa) { return (p4d_t *)__va(pa); } @@ -624,7 +622,7 @@ static phys_addr_t __init alloc_p4d_fixmap(uintptr_t va) return memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE); } -static phys_addr_t alloc_p4d_late(uintptr_t va) +static phys_addr_t __meminit alloc_p4d_late(uintptr_t va) { unsigned long vaddr; @@ -633,9 +631,8 @@ static phys_addr_t alloc_p4d_late(uintptr_t va) return __pa(vaddr); } -static void __init create_pud_mapping(pud_t *pudp, - uintptr_t va, phys_addr_t pa, - phys_addr_t sz, pgprot_t prot) +static void __meminit create_pud_mapping(pud_t *pudp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, + pgprot_t prot) { pmd_t *nextp; phys_addr_t next_phys; @@ -660,9 +657,8 @@ static void __init create_pud_mapping(pud_t *pudp, create_pmd_mapping(nextp, va, pa, sz, prot); } -static void __init create_p4d_mapping(p4d_t *p4dp, - uintptr_t va, phys_addr_t pa, - phys_addr_t sz, pgprot_t prot) +static void __meminit create_p4d_mapping(p4d_t *p4dp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, + pgprot_t prot) { pud_t *nextp; phys_addr_t next_phys; @@ -718,9 +714,8 @@ static void __init create_p4d_mapping(p4d_t *p4dp, #define create_pmd_mapping(__pmdp, __va, __pa, __sz, __prot) do {} while(0) #endif /* __PAGETABLE_PMD_FOLDED */ -void __init create_pgd_mapping(pgd_t *pgdp, - uintptr_t va, phys_addr_t pa, - phys_addr_t sz, pgprot_t prot) +void __meminit create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, + pgprot_t prot) { pgd_next_t *nextp; phys_addr_t next_phys; @@ -745,7 +740,7 @@ void __init create_pgd_mapping(pgd_t *pgdp, create_pgd_next_mapping(nextp, va, pa, sz, prot); } -static uintptr_t __init best_map_size(phys_addr_t base, phys_addr_t size) +static uintptr_t __meminit best_map_size(phys_addr_t base, phys_addr_t size) { if (!(base & (PGDIR_SIZE - 1)) && size >= PGDIR_SIZE) return PGDIR_SIZE; @@ -778,7 +773,7 @@ asmlinkage void __init __copy_data(void) #endif #ifdef CONFIG_STRICT_KERNEL_RWX -static __init pgprot_t pgprot_from_va(uintptr_t va) +static __meminit pgprot_t pgprot_from_va(uintptr_t va) { if (is_va_kernel_text(va)) return PAGE_KERNEL_READ_EXEC; @@ -805,7 +800,7 @@ void mark_rodata_ro(void) debug_checkwx(); } #else -static __init pgprot_t pgprot_from_va(uintptr_t va) +static __meminit pgprot_t pgprot_from_va(uintptr_t va) { if (IS_ENABLED(CONFIG_64BIT) && !is_kernel_mapping(va)) return PAGE_KERNEL; From patchwork Fri May 12 14:57:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 13239444 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 23D3DC7EE23 for ; Fri, 12 May 2023 14:58:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=B/sF15N3dWe0a+xh7Td3fqd7VLDDVu4X9yqnLhoYSpo=; b=DyC1shvDtq/bsa gJfsefrzjvsUHKnjYrE9A+aKbW4rx+RSxa76WgMbL6wusXkXmv34i2kD8rF4hSgC/o5Mynsi3YWGA XRBIhXb22RCF0i4GzUJkcOYuhS37TZWKohTsZZvc0pc9DKkPBTeNN6ysKoRN4xJzN2TGeXBUmmC/V JBMKH30xPtSxI75WHk1QHNPuR9ux+cGtQaQbIV+YMiuaP/93ZUgDcw3vzLhyzoQggAKX+Ys+Gp2fz 7LPCv8+mdlz4purJtf1vOMi/p7XAITOJeutav6i7BPMTTxhhf6qqH5KTMTGLPPvP1iIRQC0azrrKN HXBM1j8qJGoMMzV/xA8A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pxUDi-00CFyM-1q; Fri, 12 May 2023 14:58:10 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pxUDe-00CFvm-2Z for linux-riscv@lists.infradead.org; Fri, 12 May 2023 14:58:09 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6610C654DC; Fri, 12 May 2023 14:58:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2F83CC433D2; Fri, 12 May 2023 14:58:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683903485; bh=aaVyrjFt4Z24qTU7IkdzOXPq++vuBvyBWDRYOQbS9VA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YPQSWSrA4888qRlom2V9n0E4yFAcDlwO6kLEPA1Z3/jN+kNGhPXLeY8aFL0ESQTul RMyaIs6qUO82q50vSJrCQWrwL5FL7Zd2It0w/k09t7jeuI9/8zofE8JzaXyKCEZlUs 6vZOfEid0lLW22Wa0vb8u5t7SQjIJphqI/J9HwOCpQIywm81oMLE6fnM89OUtnUkFn j8XQGA0U7Xt+BTOHhoHC7wsFBB8N8mbrfH+4xlCjWl8xkta+DzSiBxsCwtLgzfktYM MaJ2hhSLpdhYY4k+7RXHrTEDITwtEA6t4tJs5cfx65BsquOpd3IWQp5793KvR6uCq6 t5vc9nXQmqCFQ== From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-riscv@lists.infradead.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-kernel@vger.kernel.org, linux-mm@kvack.org, David Hildenbrand , Oscar Salvador , virtualization@lists.linux-foundation.org, linux@rivosinc.com, Alexandre Ghiti Subject: [PATCH 3/7] riscv: mm: Refactor create_linear_mapping_range() for hot add Date: Fri, 12 May 2023 16:57:33 +0200 Message-Id: <20230512145737.985671-4-bjorn@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230512145737.985671-1-bjorn@kernel.org> References: <20230512145737.985671-1-bjorn@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230512_075806_869495_36B4F5BD X-CRM114-Status: GOOD ( 11.05 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Björn Töpel Add a parameter to the direct map setup function, so it can be used in arch_add_memory() later. Signed-off-by: Björn Töpel --- arch/riscv/mm/init.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index e974ff6ef036..aea8ccb3f4ae 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -1247,18 +1247,19 @@ asmlinkage void __init setup_vm(uintptr_t dtb_pa) pt_ops_set_fixmap(); } -static void __init create_linear_mapping_range(phys_addr_t start, - phys_addr_t end) +static void __meminit create_linear_mapping_range(phys_addr_t start, phys_addr_t end, + struct mhp_params *params) { phys_addr_t pa; uintptr_t va, map_size; for (pa = start; pa < end; pa += map_size) { + pgprot_t pgprot; + va = (uintptr_t)__va(pa); + pgprot = params ? params->pgprot : pgprot_from_va(va); map_size = best_map_size(pa, end - pa); - - create_pgd_mapping(swapper_pg_dir, va, pa, map_size, - pgprot_from_va(va)); + create_pgd_mapping(swapper_pg_dir, va, pa, map_size, pgprot); } } @@ -1288,13 +1289,12 @@ static void __init create_linear_mapping_page_table(void) if (end >= __pa(PAGE_OFFSET) + memory_limit) end = __pa(PAGE_OFFSET) + memory_limit; - create_linear_mapping_range(start, end); + create_linear_mapping_range(start, end, NULL); } #ifdef CONFIG_STRICT_KERNEL_RWX - create_linear_mapping_range(ktext_start, ktext_start + ktext_size); - create_linear_mapping_range(krodata_start, - krodata_start + krodata_size); + create_linear_mapping_range(ktext_start, ktext_start + ktext_size, NULL); + create_linear_mapping_range(krodata_start, krodata_start + krodata_size, NULL); memblock_clear_nomap(ktext_start, ktext_size); memblock_clear_nomap(krodata_start, krodata_size); From patchwork Fri May 12 14:57:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 13239445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BDCE5C7EE2D for ; Fri, 12 May 2023 14:58:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=P6OoKPtQz1UrD8EFihzCIMPldlQSwjM2u8jBR/vAWIw=; b=m91KZ4lvopBC0s 5Bds22F/Q8AOZng5TWk2HkVBVDbnbe3V8qqfJn9VRhnaiAnk/iX7krSH1vZLfEgFRg+v5ZD4zBNoo w/t5Ou9uM4vISw7+F1eTtBsR+O08AwznvGxnX8IRMTswgfL4SjZDPl2UWptaZkDsgNM8PBpYvHUiV W6fOIhKpwUT6V+t5s+4lk5oKi4N0O5aaDYYBM1PHW9wWbt4i856ExewFEJEtqVGlBMJyG5EliEcyH tvepc6T9tHV4ur2mmddp1EXIUfVbFmqDYdHDUFKTPvFXV96BiL6LLoritn53OVOE7TU7hiMMcW9CH 7nxLCdTFszaX1lN4OzZQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pxUDl-00CG09-0i; Fri, 12 May 2023 14:58:13 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pxUDi-00CFxe-0T for linux-riscv@lists.infradead.org; Fri, 12 May 2023 14:58:11 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7BE6C6133B; Fri, 12 May 2023 14:58:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 414DCC433EF; Fri, 12 May 2023 14:58:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683903488; bh=2Hg7N6ebdPoIJZTj/Z2XT0CsZmR9YcBL0bRVObnrrco=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Os5JiE+94pc+yBVvOpqjMSDiGSPKKDvcZURYVV7tm3zq/Llfx7r+svdbETNcWHa9V Li9vVWnnYPg4WmY1beATZukfwTwxRLiQJpQmEs/GjF49Mf4C7W1sm0VTzGAs3LvJ4V fiW7zPjP32r2q8vcrAiansu0eVnkhKadKCYOX+a3UFnS4f2mBOnAYzv6slNn4OKi0T BBktSF+bxWWWpHNDAtW4wOAZ8gQ/LY6hbDNeK7wW+z40xpd+VuFNYevwb5gOiYtXJB 0A/zrgC1iRCSbGLV+DiScyv8mJHKg3G9Tak/03DMMiegzuW6pG8j5kuNpkGbWaRLoJ nuEOcUj9yW5Aw== From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-riscv@lists.infradead.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-kernel@vger.kernel.org, linux-mm@kvack.org, David Hildenbrand , Oscar Salvador , virtualization@lists.linux-foundation.org, linux@rivosinc.com, Alexandre Ghiti Subject: [PATCH 4/7] riscv: mm: Add memory hot add/remove support Date: Fri, 12 May 2023 16:57:34 +0200 Message-Id: <20230512145737.985671-5-bjorn@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230512145737.985671-1-bjorn@kernel.org> References: <20230512145737.985671-1-bjorn@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230512_075810_268732_938D9CDF X-CRM114-Status: GOOD ( 17.88 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Björn Töpel From an arch perspective, a couple of callbacks needs to be implemented to support hotplugging: arch_add_memory() This callback is responsible for updating the linear/direct map, and call into the memory hotplugging generic code via __add_pages(). arch_remove_memory() In this callback the linear/direct map is tore down. vmemmap_free() The function tears down the vmemmap mappings (if CONFIG_SPARSEMEM_VMEMMAP is in-use), and also deallocates the backing vmemmap pages. Note that for persistent memory, an alternative allocator for the backing pages can be used -- the vmem_altmap. This means that when the backing pages are cleared, extra care is needed so that the correct deallocation method is used. Note that RISC-V populates the vmemmap using vmemmap_populate_basepages(), so currently no hugepages are used for the backing store. The page table unmap/teardown functions are heavily based (copied!) from the x86 tree. The same remove_pgd_mapping() is used in both vmemmap_free() and arch_remove_memory(), but in the latter function the backing pages are not removed. Signed-off-by: Björn Töpel --- arch/riscv/mm/init.c | 233 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 233 insertions(+) diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index aea8ccb3f4ae..a468708d1e1c 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -1444,3 +1444,236 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, return vmemmap_populate_basepages(start, end, node, NULL); } #endif + +#ifdef CONFIG_MEMORY_HOTPLUG +static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd) +{ + pte_t *pte; + int i; + + for (i = 0; i < PTRS_PER_PTE; i++) { + pte = pte_start + i; + if (!pte_none(*pte)) + return; + } + + free_pages((unsigned long)page_address(pmd_page(*pmd)), 0); + pmd_clear(pmd); +} + +static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud) +{ + pmd_t *pmd; + int i; + + for (i = 0; i < PTRS_PER_PMD; i++) { + pmd = pmd_start + i; + if (!pmd_none(*pmd)) + return; + } + + free_pages((unsigned long)page_address(pud_page(*pud)), 0); + pud_clear(pud); +} + +static void __meminit free_pud_table(pud_t *pud_start, p4d_t *p4d) +{ + pud_t *pud; + int i; + + for (i = 0; i < PTRS_PER_PUD; i++) { + pud = pud_start + i; + if (!pud_none(*pud)) + return; + } + + free_pages((unsigned long)page_address(p4d_page(*p4d)), 0); + p4d_clear(p4d); +} + +static void __meminit free_vmemmap_storage(struct page *page, size_t size, + struct vmem_altmap *altmap) +{ + if (altmap) + vmem_altmap_free(altmap, size >> PAGE_SHIFT); + else + free_pages((unsigned long)page_address(page), get_order(size)); +} + +static void __meminit remove_pte_mapping(pte_t *pte_base, unsigned long addr, unsigned long end, + bool is_vmemmap, struct vmem_altmap *altmap) +{ + unsigned long next; + pte_t *ptep, pte; + + for (; addr < end; addr = next) { + next = (addr + PAGE_SIZE) & PAGE_MASK; + if (next > end) + next = end; + + ptep = pte_base + pte_index(addr); + pte = READ_ONCE(*ptep); + + if (!pte_present(*ptep)) + continue; + + pte_clear(&init_mm, addr, ptep); + if (is_vmemmap) + free_vmemmap_storage(pte_page(pte), PAGE_SIZE, altmap); + } +} + +static void __meminit remove_pmd_mapping(pmd_t *pmd_base, unsigned long addr, unsigned long end, + bool is_vmemmap, struct vmem_altmap *altmap) +{ + unsigned long next; + pte_t *pte_base; + pmd_t *pmdp, pmd; + + for (; addr < end; addr = next) { + next = pmd_addr_end(addr, end); + pmdp = pmd_base + pmd_index(addr); + pmd = READ_ONCE(*pmdp); + + if (!pmd_present(pmd)) + continue; + + if (pmd_leaf(pmd)) { + pmd_clear(pmdp); + if (is_vmemmap) + free_vmemmap_storage(pmd_page(pmd), PMD_SIZE, altmap); + continue; + } + + pte_base = (pte_t *)pmd_page_vaddr(*pmdp); + remove_pte_mapping(pte_base, addr, next, is_vmemmap, altmap); + free_pte_table(pte_base, pmdp); + } +} + +static void __meminit remove_pud_mapping(pud_t *pud_base, unsigned long addr, unsigned long end, + bool is_vmemmap, struct vmem_altmap *altmap) +{ + unsigned long next; + pud_t *pudp, pud; + pmd_t *pmd_base; + + for (; addr < end; addr = next) { + next = pud_addr_end(addr, end); + pudp = pud_base + pud_index(addr); + pud = READ_ONCE(*pudp); + + if (!pud_present(pud)) + continue; + + if (pud_leaf(pud)) { + if (pgtable_l4_enabled) { + pud_clear(pudp); + if (is_vmemmap) + free_vmemmap_storage(pud_page(pud), PUD_SIZE, altmap); + } + continue; + } + + pmd_base = pmd_offset(pudp, 0); + remove_pmd_mapping(pmd_base, addr, next, is_vmemmap, altmap); + + if (pgtable_l4_enabled) + free_pmd_table(pmd_base, pudp); + } +} + +static void __meminit remove_p4d_mapping(p4d_t *p4d_base, unsigned long addr, unsigned long end, + bool is_vmemmap, struct vmem_altmap *altmap) +{ + unsigned long next; + p4d_t *p4dp, p4d; + pud_t *pud_base; + + for (; addr < end; addr = next) { + next = p4d_addr_end(addr, end); + p4dp = p4d_base + p4d_index(addr); + p4d = READ_ONCE(*p4dp); + + if (!p4d_present(p4d)) + continue; + + if (p4d_leaf(p4d)) { + if (pgtable_l5_enabled) { + p4d_clear(p4dp); + if (is_vmemmap) + free_vmemmap_storage(p4d_page(p4d), P4D_SIZE, altmap); + } + continue; + } + + pud_base = pud_offset(p4dp, 0); + remove_pud_mapping(pud_base, addr, next, is_vmemmap, altmap); + + if (pgtable_l5_enabled) + free_pud_table(pud_base, p4dp); + } +} + +static void __meminit remove_pgd_mapping(unsigned long va, unsigned long end, bool is_vmemmap, + struct vmem_altmap *altmap) +{ + unsigned long addr, next; + p4d_t *p4d_base; + pgd_t *pgd; + + for (addr = va; addr < end; addr = next) { + next = pgd_addr_end(addr, end); + pgd = pgd_offset_k(addr); + + if (!pgd_present(*pgd)) + continue; + + if (pgd_leaf(*pgd)) + continue; + + p4d_base = p4d_offset(pgd, 0); + remove_p4d_mapping(p4d_base, addr, next, is_vmemmap, altmap); + } + + flush_tlb_all(); +} + +static void __meminit remove_linear_mapping(phys_addr_t start, u64 size) +{ + unsigned long va = (unsigned long)__va(start); + unsigned long end = (unsigned long)__va(start + size); + + remove_pgd_mapping(va, end, false, NULL); +} + +int __ref arch_add_memory(int nid, u64 start, u64 size, struct mhp_params *params) +{ + int ret; + + create_linear_mapping_range(start, start + size, params); + flush_tlb_all(); + ret = __add_pages(nid, start >> PAGE_SHIFT, size >> PAGE_SHIFT, params); + if (ret) { + remove_linear_mapping(start, size); + return ret; + } + + max_pfn = PFN_UP(start + size); + max_low_pfn = max_pfn; + return 0; +} + +void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap) +{ + __remove_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT, altmap); + remove_linear_mapping(start, size); +} + +#ifdef CONFIG_SPARSEMEM_VMEMMAP +void __ref vmemmap_free(unsigned long start, unsigned long end, struct vmem_altmap *altmap) +{ + remove_pgd_mapping(start, end, true, altmap); +} +#endif /* CONFIG_SPARSEMEM_VMEMMAP */ +#endif /* CONFIG_MEMORY_HOTPLUG */ From patchwork Fri May 12 14:57:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 13239446 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D229BC7EE2F for ; Fri, 12 May 2023 14:58:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=s+/Nxq4flHPkpmJLOBkq+gR4xlWaDmYX3NXHP9DPJ5A=; b=UHNfvp3d7JobCI ZfP4uX8FMMtajY1J2/S0YLjXkR3NRrs/AwRyyGbvpgMdEgX1DY5X1VmMX65UIRNoCz0r8hjCOysEp qNHzaEjBVx2OIEohO656WJIh/utzjaqnvw1X2v+qlb6FPGr4ZFShNgKj3Zs+BiJRZvaewXKtDg6Tg Dd4Rwa9j/lMZ9G5NDnvhr0EzAmbMW9b3mZzVHiN7qVnWPebnAWpx6pxDkXV3onyW3MQDGtNqIpHuR lLu5G9WIJ1hCgPzF2o0b1nBpX6EisR+bFNNNYy5lBXFUCHBD2jQsaYCffqKkVEeiUioj5LCU3jEgC yXzfN46fdBcx1nRUXuZg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pxUDn-00CG26-2g; Fri, 12 May 2023 14:58:15 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pxUDk-00CFzd-2S for linux-riscv@lists.infradead.org; Fri, 12 May 2023 14:58:13 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 573A061137; Fri, 12 May 2023 14:58:12 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5593AC4339E; Fri, 12 May 2023 14:58:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683903492; bh=Pna6hgXZPMsGHenSDkIIzlTOUDQrq5sDRKKv7+K0xpY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qQzflYqn6X1XwTYqDskdaUvkozpqmlKiTZ7sJZMcSXHaRn188NF1w4O4Gcg9IS09k oKwEhmh46ngeDtbZCpX71wFNrPl+kK3Y4p+ewFov1/UkgDjuFqVOTty0tiMMSElTNb yMuyw+QZmlU6JyZOeqC3QUttWJyV6GkHm73I62awiyhzXWWbPf3Lm1PjNuZoNrq0QB VApf+wgzHLHZ2xAEvvSCNJwzRHGvl/v5NT9pDPzmfOPYPJjDHZOW346ZF/PUDBnppQ xqqHnvoiJdxwAchWZf8t4aSIlDXEgYY06mtzf3klWqYHbjBSQG5Jhc3+1nVtzW5hkE X+9KbCvtUB98A== From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-riscv@lists.infradead.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-kernel@vger.kernel.org, linux-mm@kvack.org, David Hildenbrand , Oscar Salvador , virtualization@lists.linux-foundation.org, linux@rivosinc.com, Alexandre Ghiti Subject: [PATCH 5/7] riscv: Enable memory hot add/remove arch kbuild support Date: Fri, 12 May 2023 16:57:35 +0200 Message-Id: <20230512145737.985671-6-bjorn@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230512145737.985671-1-bjorn@kernel.org> References: <20230512145737.985671-1-bjorn@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230512_075812_835849_4D683CCA X-CRM114-Status: UNSURE ( 9.23 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Björn Töpel Enable ARCH_ENABLE_MEMORY_HOTPLUG and ARCH_ENABLE_MEMORY_HOTREMOVE for RISC-V. Signed-off-by: Björn Töpel --- arch/riscv/Kconfig | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 348c0fa1fc8c..81b3f188f396 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -14,6 +14,8 @@ config RISCV def_bool y select ARCH_DMA_DEFAULT_COHERENT select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION + select ARCH_ENABLE_MEMORY_HOTPLUG if SPARSEMEM && 64BIT && MMU + select ARCH_ENABLE_MEMORY_HOTREMOVE if SPARSEMEM && 64BIT && MMU select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE select ARCH_HAS_BINFMT_FLAT From patchwork Fri May 12 14:57:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 13239447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 586A6C7EE23 for ; Fri, 12 May 2023 14:58:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=LwjhJe7PVjKk3EeO9qNaIVzyuxKu/JaEQkKV7ME+3Xg=; b=sRFKZC/QdqY+En tkerRYV9zbQs2DLA6aA+elNU5ZfS/j6R1a6XIxJPBit14AEhi3uv2J2/Luxb+LU+LkPjMZia/U8QM MK8pRQDk5VMLtr6rR9DOlFOomY8Zi8M6CexzlAgGPu/5aE7sm2DzmEIrpinYKdMGTckOnXboDPva8 T3TTfjKCY81AV0FeBymd7KvlNpHCuSEmA+ybeXEkQGoTmTqhbC10A3xJLLzmY6dPYHFRzYnYdWFLR tNnlCcHcm4Au2w4WcRaUJYicMNC5NHwq+2FCqS+gAerfLb6FWLKIl4/h4PxtwdTm8DDm87CNXkLUz 5Pqv59cpRIeUX7FZw1dw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pxUDr-00CG4U-1e; Fri, 12 May 2023 14:58:19 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pxUDo-00CG2V-1Q for linux-riscv@lists.infradead.org; Fri, 12 May 2023 14:58:18 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 06B8965766; Fri, 12 May 2023 14:58:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AF583C4339B; Fri, 12 May 2023 14:58:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683903495; bh=OlUUIHh8F/ikEn4ruOXdEkA3wElniYzLmKIr7MhQ0Hw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NcLbHLWxqI2bumvNceUzMAPv+C0Y746k1Jp/7tXA1GgJqday/8MB/23BL0VcKD2EF l7sOoE8BAouxMF63X6G55suALJoO0H7hzI5XVSs9OXLwTPoBkriEnruif/KXQ3aW+/ SFMo23v3zgADZZc+8aPweKC+kX6VdWxTcQh0IOUvIyRD43+WepPIyPmdzGCt6wUJWx R49ozGhEMEXMJZEifDbl4mVjzatCqLEVcObGNbLsH/wJ2aox1/STvfILMriazGRm51 mnOGBU56o6Scmd5rVB7pPYs1z+zlGd1OdAwUcUAL4VkEWtJlONEsexD186+SZnZ1pr dXjChETIBrCjA== From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-riscv@lists.infradead.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-kernel@vger.kernel.org, linux-mm@kvack.org, David Hildenbrand , Oscar Salvador , virtualization@lists.linux-foundation.org, linux@rivosinc.com, Alexandre Ghiti Subject: [PATCH 6/7] virtio-mem: Enable virtio-mem for RISC-V Date: Fri, 12 May 2023 16:57:36 +0200 Message-Id: <20230512145737.985671-7-bjorn@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230512145737.985671-1-bjorn@kernel.org> References: <20230512145737.985671-1-bjorn@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230512_075816_517285_24839A14 X-CRM114-Status: GOOD ( 11.02 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Björn Töpel Now that RISC-V has memory hot add/remove support, virtio-mem can be used on the platform. Signed-off-by: Björn Töpel --- drivers/virtio/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig index 0a53a61231c2..358e79ece169 100644 --- a/drivers/virtio/Kconfig +++ b/drivers/virtio/Kconfig @@ -117,7 +117,7 @@ config VIRTIO_BALLOON config VIRTIO_MEM tristate "Virtio mem driver" - depends on X86_64 || ARM64 + depends on X86_64 || ARM64 || RISCV depends on VIRTIO depends on MEMORY_HOTPLUG depends on MEMORY_HOTREMOVE From patchwork Fri May 12 14:57:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 13239448 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 75443C77B7C for ; Fri, 12 May 2023 14:58:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=waFDC0FctqamwwemQfiHMj4mwT4LRaQl2De1HM7vwfI=; b=AjwCPNxa6W6j2t uc1A1G5Q1WXfGbgBIWJX2DZZXoKqoIWzKbI2/44QHfe6LW6jvqUvrk7fwjonxP3qJ67q18nPt4fR8 x6MvRKCwBQXlyeJYLyvmGj2HFLTgk7icUTGzKW/YPANbl99/2JPY90KiZhhJUaxcCVe5PPoxi+lLW CSv/WHQmZfO3br9WBwUKpml0+X+9cvSNQ6vV6yAQSsQBXqGa3jlNbeMjqEIdv/c418LmMBZQKjiWn nFD37rHBZBIdz4eudeO7eFSAaH1dfET8CAFvPzuRVYIy/6N0ngH/sq56C4RbqgnfZx+cUKSyBy1mf HsqsbUSKXBaL2Ez07Iig==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pxUDu-00CG6R-0c; Fri, 12 May 2023 14:58:22 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pxUDr-00CG4R-2O for linux-riscv@lists.infradead.org; Fri, 12 May 2023 14:58:21 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4735E655F5; Fri, 12 May 2023 14:58:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E9BF1C4339C; Fri, 12 May 2023 14:58:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683903498; bh=lRwO3YfIMF9fmZSTE+/WO7BqWpiO2x+8QNF4La5hiEE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lFJE9ofAvx/W7GMthxymGs+b9O5oFO1BwdKIGcHaAN62YeWePDQ0N/sK71knTdSoF FVBIIqU4AmmF6IG79I2XtGg0JavzeJwFrHvwtyzSV76oVsE2HRceOMMnAX1c65IAP/ wzRAeUAfR4ibQ4s6pSnjOcUPEJrI0Uwkbd762zC/EYqhScoqAKaEuy84NEiMva3S8O cUjmxdLyvrJYLCFOB3H2n8gsVJcGT3aBKjaVsXZO/B0iymi2R+/sIoE+4yZ4U/slfd k0/FpzS+nNiLNCLb4BGq3Qd2jSem/r0rZlEnE2QEYIKn4OVGCS3P4sJXIb6FK2umT0 TTdE0MNOfw90w== From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-riscv@lists.infradead.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-kernel@vger.kernel.org, linux-mm@kvack.org, David Hildenbrand , Oscar Salvador , virtualization@lists.linux-foundation.org, linux@rivosinc.com, Alexandre Ghiti Subject: [PATCH 7/7] riscv: mm: Pre-allocate vmalloc PGD leaves Date: Fri, 12 May 2023 16:57:37 +0200 Message-Id: <20230512145737.985671-8-bjorn@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230512145737.985671-1-bjorn@kernel.org> References: <20230512145737.985671-1-bjorn@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230512_075819_825597_8784DD5B X-CRM114-Status: GOOD ( 14.09 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Björn Töpel Instead of relying on vmalloc_fault() to synchronize the page-tables, pre-allocate the PGD leaves of the vmalloc area. This is only enabled if memory hot/add is enabled by the build. Signed-off-by: Björn Töpel --- arch/riscv/mm/fault.c | 7 ++++++- arch/riscv/mm/init.c | 1 + 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index 8685f85a7474..b61e279acd50 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -233,12 +233,17 @@ void handle_page_fault(struct pt_regs *regs) * Fault-in kernel-space virtual memory on-demand. * The 'reference' page table is init_mm.pgd. * + * For memory hotplug enabled systems, the PGD entries are + * pre-allocated, which avoids the need to synchronize + * pgd/fault-in. + * * NOTE! We MUST NOT take any locks for this case. We may * be in an interrupt or a critical region, and should * only copy the information from the master page table, * nothing more. */ - if (unlikely((addr >= VMALLOC_START) && (addr < VMALLOC_END))) { + if (unlikely(!IS_ENABLED(CONFIG_MEMORY_HOTPLUG) && + (addr >= VMALLOC_START) && (addr < VMALLOC_END))) { vmalloc_fault(regs, code, addr); return; } diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index a468708d1e1c..fd5a6d3fe182 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -236,6 +236,7 @@ static void __init preallocate_pgd_pages_range(unsigned long start, unsigned lon static void __init prepare_memory_hotplug(void) { #ifdef CONFIG_MEMORY_HOTPLUG + preallocate_pgd_pages_range(VMALLOC_START, VMALLOC_END, "vmalloc"); preallocate_pgd_pages_range(VMEMMAP_START, VMEMMAP_END, "vmemmap"); preallocate_pgd_pages_range(PAGE_OFFSET, PAGE_END, "direct map"); #endif