From patchwork Wed May 31 09:38:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 13261843 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8915FC77B7C for ; Wed, 31 May 2023 09:38:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BF7638E0002; Wed, 31 May 2023 05:38:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BA6D68E0001; Wed, 31 May 2023 05:38:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A95728E0002; Wed, 31 May 2023 05:38:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 985958E0001 for ; Wed, 31 May 2023 05:38:27 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 5D3C71402E8 for ; Wed, 31 May 2023 09:38:27 +0000 (UTC) X-FDA: 80850049854.07.E017088 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf13.hostedemail.com (Postfix) with ESMTP id 9A3E820010 for ; Wed, 31 May 2023 09:38:25 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=FILT1twb; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf13.hostedemail.com: domain of bjorn@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bjorn@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685525905; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=gx6zf4P2YY1iSC1efCID0MZjLNpSTglDBrpoPrUNmfY=; b=GZg31zKMvENDtBBvXQichI30r5GjkS6u5A7vbrbmRA5I7ZxywDJW3rtLKluVyGG8txICM2 IMdRPXPZhqC6mxnGIRAmx9QcWunQuTobVHzD6V9pPIixEbwDRSL3RcJODzDe1a9T4LZMkq EvnmZy8nIwNRqBvFGY53H4JASoM87OI= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=FILT1twb; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf13.hostedemail.com: domain of bjorn@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bjorn@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685525905; a=rsa-sha256; cv=none; b=tSO7p8kdlJ+KQnLZ/6R6hlrR6ABtFIr33bAye+SeAk4K4iIHMYvgzm8se8WypYFfQsT30T YpbErWPj8s6hL/VLeOF0/Hog60coMpMx/eVudv8xqxMh1FMXlU48a68OGcV99niq8dDztp WDQY6L4UdisaNJ8j2si3QUZ9/bWF54k= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 85AA462D99; Wed, 31 May 2023 09:38:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 07B3FC433D2; Wed, 31 May 2023 09:38:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685525904; bh=wqDGhSnnNiEHWdcXEiKZCvqzE4oLjPYSd88Km/Hcc50=; h=From:To:Cc:Subject:Date:From; b=FILT1twbdRZSFUQEanPkdAm4xeTw49ij5Wuy1k3hR7TouHmtkKXuGKRaiF4LGGiWa zra8PeXcM6EZnbcWhTyPeV+kzAZx8g4FXZvxHoF7ODOYjhRn8U8xPBxEkP3xk57xli hWr0dW1S4J8foOfk9FcWiIDAYz2vo5PgOInK53WBlwdUCftdjsugy9huLvga1XACSS M52X949hemUp6U/GIwhRUYkoZWOdweF0eIif2AMntEALGkuhhdHycPiJBU21PuJ7De WR7P7DxUT44nTy1cNQCUB3dLJZ6+BGf4XHb2NFrJHE55A6kmFJc24TrRPvbVAZ01RQ V3YhHSsIG5Qig== From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-riscv@lists.infradead.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux@rivosinc.com, Alexandre Ghiti , Joerg Roedel Subject: [PATCH v2] riscv: mm: Pre-allocate PGD entries for vmalloc/modules area Date: Wed, 31 May 2023 11:38:17 +0200 Message-Id: <20230531093817.665799-1-bjorn@kernel.org> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-Rspamd-Queue-Id: 9A3E820010 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 943ag3r53bhdui1cqehutq15weefktyr X-HE-Tag: 1685525905-599128 X-HE-Meta: U2FsdGVkX19I8cbqbsOz+Hao1j2D+qeYjcDig3KsJXU7YdidChg+LQokZ9LMe8TyEP00Jyew8+mlcLJi0TghVzEoGAUh6NYKOeGg+ngcXfvlM/EX7i3s6kMFCWWBqIyfY5x/sa20tO9FhiLUDlfZyI+IgyjDyNcyqpxjSDF82P8dsomVRnpD5dJFxp08gIuxwob+kx+v2tUf/PD7OUpLCdcYOUpg2uN6kzR4AMtPYcZG4ioFWaqP8TD1XXvy4X2u0H1cmqKA+UXMwwtTF0fqD4ouuoGDZ+L0j86aI5CTPzpsvVE/bcDBqFxXQesO2dmGRZxx3gIZdgEAOezFEejQ5dhXSQYxiOZaWp+w3zTXKum3FfSr9pCITGeIIe8co8vpPW7G8dbvDti6+uj8NHoWnD+SaCFmKFWhTZ7yIjzXJTAWdTK/Vk+s6zxQyPqPJN1Eq491WWNQrqb8LMvfZuHMWCD4W45slfUEr02Gr0uKPWNXGZ2kFxgjtr0WYcXouOAVoxTe2JM3RX2Sq3KBX+fJ+e7ZnlaV5NCd29FJw1PGMtMcn3VZ3dZdbbjYPvROFhpQ0Y0ne/oQm7ahDIrKeGsr0wkKr8IP0r7jKhxTe9FJdjNxQ5IQ91YSU9Dd+97EDf256OqspWhLixi8mTgqOyDWJ/nRM3DRRWRGYeazTTTJm8ofb2jxVwSf8tbv6Fq42H2NoVBh4yTvHtwfj+N7T+rlMO3MXNpmlqS0DOvhxBM+V/a00ZTqPokZCLxMeYoPBqqFCnhalVhevUbXQuPwP5NSxdyiC9w1XjqnCJwYgSJ6GK16nCbu/Zajeouzu/h5CCJiDZj5+3aKMLnXXnWobwIP5vcVAkPi86OhP+XRpN/DH9MeWwJAO0PUP6qndSj1cPRsTRaTjQkh1Qo1wTlS1y4RWSCbrupCsz+5WeQ8BnJaatrrlInPYHD1vgiq3aeQ5tWzI04aGshV5VfwrdoncX4 PJaw6ov2 /b9vGyNUSenM3iqRL2fE9dY3hXKLXq3rxoIBY121o105R6Yopx24DoqBCBJkxOO7051URcvFQ4uJSRVZmUcfnzFm2BqpUmXoGgYb2MFz9eemHSxye6J72/EUuQfwwcNtdKreIWYx+yikpXCuxiYlmLYDgO/vhjAlMXj9OPjK9HxQSyhBEsNhbUPRmbA2KEhJ2iIJW9W0mEw7jlt1E7ShL4Owc5TnP/n/A0jcTrg0yBZ+KBtshd4oRn4vJCrSDAgEvYZ6mmD1s47EjaJpd32Ec0LSH1HeBWDuxu9ze/e0dK1zVc/Kh+4FQ0naNjUHFEymWlv06K64FUpS8xqmqinVbNdkbrQVFKhk5JkPPjf2QIkoRMz8abqEVyb8K+KsAfkftPIDPE6Qc8DCWNn6O+q918qvxv4Z/11PP+7ME4O8GiJvus6G+e1pfF9chATS+UNXyjssX4wosVI7V719PyqO1BM4227SQQh6OBAABLRG+MZtjRBeyWWUkvFKf6knihbgAj9Yg0m8d6z0o//tO1CXsFSK2nvMJ3uf/6XZQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Björn Töpel The RISC-V port requires that kernel PGD entries are to be synchronized between MMs. This is done via the vmalloc_fault() function, that simply copies the PGD entries from init_mm to the faulting one. Historically, faulting in PGD entries have been a source for both bugs [1], and poor performance. One way to get rid of vmalloc faults is by pre-allocating the PGD entries. Pre-allocating the entries potientially wastes 64 * 4K (65 on SV39). The pre-allocation function is pulled from Jörg Rödel's x86 work, with the addition of 3-level page tables (PMD allocations). The pmd_alloc() function needs the ptlock cache to be initialized (when split page locks is enabled), so the pre-allocation is done in a RISC-V specific pgtable_cache_init() implementation. Pre-allocate the kernel PGD entries for the vmalloc/modules area, but only for 64b platforms. Link: https://lore.kernel.org/lkml/20200508144043.13893-1-joro@8bytes.org/ # [1] Signed-off-by: Björn Töpel Reviewed-by: Alexandre Ghiti --- v1->v2: Fixed broken !MMU build. --- arch/riscv/mm/fault.c | 16 ++---------- arch/riscv/mm/init.c | 58 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+), 14 deletions(-) base-commit: ac9a78681b921877518763ba0e89202254349d1b diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index 8685f85a7474..b023fb311e28 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -238,24 +238,12 @@ void handle_page_fault(struct pt_regs *regs) * only copy the information from the master page table, * nothing more. */ - if (unlikely((addr >= VMALLOC_START) && (addr < VMALLOC_END))) { + if ((!IS_ENABLED(CONFIG_MMU) || !IS_ENABLED(CONFIG_64BIT)) && + unlikely(addr >= VMALLOC_START && addr < VMALLOC_END)) { vmalloc_fault(regs, code, addr); return; } -#ifdef CONFIG_64BIT - /* - * Modules in 64bit kernels lie in their own virtual region which is not - * in the vmalloc region, but dealing with page faults in this region - * or the vmalloc region amounts to doing the same thing: checking that - * the mapping exists in init_mm.pgd and updating user page table, so - * just use vmalloc_fault. - */ - if (unlikely(addr >= MODULES_VADDR && addr < MODULES_END)) { - vmalloc_fault(regs, code, addr); - return; - } -#endif /* Enable interrupts if they were enabled in the parent context. */ if (!regs_irqs_disabled(regs)) local_irq_enable(); diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 747e5b1ef02d..45ceaff5679e 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -1363,3 +1363,61 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, return vmemmap_populate_basepages(start, end, node, NULL); } #endif + +#if defined(CONFIG_MMU) && defined(CONFIG_64BIT) +/* + * Pre-allocates page-table pages for a specific area in the kernel + * page-table. Only the level which needs to be synchronized between + * all page-tables is allocated because the synchronization can be + * expensive. + */ +static void __init preallocate_pgd_pages_range(unsigned long start, unsigned long end, + const char *area) +{ + unsigned long addr; + const char *lvl; + + for (addr = start; addr < end && addr >= start; addr = ALIGN(addr + 1, PGDIR_SIZE)) { + pgd_t *pgd = pgd_offset_k(addr); + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + + lvl = "p4d"; + p4d = p4d_alloc(&init_mm, pgd, addr); + if (!p4d) + goto failed; + + if (pgtable_l5_enabled) + continue; + + lvl = "pud"; + pud = pud_alloc(&init_mm, p4d, addr); + if (!pud) + goto failed; + + if (pgtable_l4_enabled) + continue; + + lvl = "pmd"; + pmd = pmd_alloc(&init_mm, pud, addr); + if (!pmd) + goto failed; + } + return; + +failed: + /* + * The pages have to be there now or they will be missing in + * process page-tables later. + */ + panic("Failed to pre-allocate %s pages for %s area\n", lvl, area); +} + +void __init pgtable_cache_init(void) +{ + preallocate_pgd_pages_range(VMALLOC_START, VMALLOC_END, "vmalloc"); + if (IS_ENABLED(CONFIG_MODULES)) + preallocate_pgd_pages_range(MODULES_VADDR, MODULES_END, "bpf/modules"); +} +#endif