From patchwork Tue Apr 8 09:52:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 14042591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3FBB2C369A4 for ; Tue, 8 Apr 2025 09:53:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4363C28000B; Tue, 8 Apr 2025 05:53:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 40C99280001; Tue, 8 Apr 2025 05:53:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D3D328000B; Tue, 8 Apr 2025 05:53:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0D2FB280001 for ; Tue, 8 Apr 2025 05:53:55 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 49AEA1A09FC for ; Tue, 8 Apr 2025 09:53:55 +0000 (UTC) X-FDA: 83310415230.07.AFA7A0A Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf23.hostedemail.com (Postfix) with ESMTP id 8797A140003 for ; Tue, 8 Apr 2025 09:53:53 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf23.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744106033; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=k9E8JC26wdK4bPf2nrNh902XnvK5CWhMXm2sdNweAJc=; b=y3iGYDwm7oVLSYtx3w+HQbKzVXdHKSG3Q6Gv0Suyo8Ep6Wj4IMD0QwFDS3HQ+AwBN5waq4 pyqOaKnvJCudoOFn6LHwFLRXP/dFxZTh/wdaaKSf349ShMoEjh9CAVk45qM9GV3F4vzHil MEWXBADBle+C2x2lghkcrkWugfwsVnM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744106033; a=rsa-sha256; cv=none; b=l/fs/0ge1ja41N5dvKCxfgHnlCe3+Om+uekUv+Nmz0dMjLCxTadMPUmvpZtYYDOJtFYikw TGJFYwbY28giuH3cnIGrTuSlkRCNQP80E1k+UtM5xaEFi/KMQXP1Av6kq1JGhhBv1QQRj4 jCcLJ0fO4otxAVZa5zC3ybBhABaq1VY= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf23.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A4D9F1691; Tue, 8 Apr 2025 02:53:53 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 64C983F6A8; Tue, 8 Apr 2025 02:53:48 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Albert Ou , Andreas Larsson , Andrew Morton , Catalin Marinas , Dave Hansen , "David S. Miller" , Geert Uytterhoeven , Linus Walleij , Madhavan Srinivasan , Mark Rutland , Matthew Wilcox , Michael Ellerman , "Mike Rapoport (IBM)" , Palmer Dabbelt , Paul Walmsley , Peter Zijlstra , Qi Zheng , Ryan Roberts , Will Deacon , Yang Shi , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-openrisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, x86@kernel.org Subject: [PATCH v2 09/12] arm64: mm: Always call PTE/PMD ctor in __create_pgd_mapping() Date: Tue, 8 Apr 2025 10:52:19 +0100 Message-ID: <20250408095222.860601-10-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250408095222.860601-1-kevin.brodsky@arm.com> References: <20250408095222.860601-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 8797A140003 X-Stat-Signature: xtsi31g41geca8azaq4i5tj9icrrewwj X-Rspam-User: X-HE-Tag: 1744106033-955803 X-HE-Meta: U2FsdGVkX19XcglI7DmJLDSUJLGzg5WxnQBNsgtkf13/BB4RdI47XPGSbNdhafIBkQ7ualy/2Ale1gvIASIfih/Pj0yjbuawho7cHrl45ZANYOu87G9XZCF4o5PfFHap7jNku3Z5ZcRrYAvU0S4sAkMeUEt9OqOaGKaMhJo37Tq0KR2Nu/2KABD1/O2AxxrpX6uO6pylNLwA5tX2vEGcF9roStU2Z+jX/PPr0Xz+qkKEQDCss7jrCu+fUYvQLkA4sy05L1Z6liCpd0SQJgS5fJiqEHoHzBJkyAet8QgfbUVSPI87H4kMAUK5PbS/1c59EzqOPM7IlOP5NLtWo+LkVCEn769iVaB/1wKJIuYiXySBQUClXqzsNvPCFfh/tbuXGvOOkVI4FqvWUotL0gmLSb+shIgpwMA0EZnpNI6tGHcSrOiq1LVt5orrrVezbPYKtj4SkcKeEP9p4Fe+oPLeK8X6hsR7yXNu4SixnwFbUWwyaTjuL1+WIxTjDJKRvNLz/f9+IezupK8R6DiPnc5SBLdNfpURUvFXSil/QGqEC46cDEsIQJ4BUTl77zidc7FN7IlyQ1KKor7+8ct2ZnpbHIhBecFpk9ZJCjDsw4OQ+0csSS/KLaSXLEoidpi8rAOyXtHub7+wUULC5v6wwTxbezAU9VCacOJxGiejHE9lUubBgAFUQ7d8vH6qJSiU+QZpblMywK3S1tGOQKmnbbCZmGvIsVm/9attV2gPRPlf6Q3ld//ccd8dnnAWwBomxSiWSWltq0ZpR3as+LGibQgv7Ajo3UoufgTvhkXojCx6SizPgw2/0/8UZMxnbS5af+zdL/R13YV+EAM4lWWafNClLtp2dsbgyijBNDzjlZUlPOZLfYPwN0IWTpxWf93vnXMrKHtX6bebPFeIJcaCC4d+4qsXvYMjLy0dhL2jkOl4TQmuwZ6GYF11WTb9a3deAu6Actbo9ag3MVTbDNnGK24 VYUzR8Ph 8tq0o/8XAQiu5B5mI7f/nvf+VHet9J844dwiTWIfjUXNVcLvRw0DZ9dDs+XIVQPdxTkU/rlRdvNVubdgmUy0oXM8+Ey/Fp0nKmFjyPqleWuaejBskIV9cK32zTyW3AwZKQrMsU/rmIV68CsL9GpTJ/6SzsjbFVY/vrbb52ipRPwA1iEPaiwj8Jk1362QnW4EUURIvOiV/HiqbU+aJlzbR88ApT7Dc+EwYY8g2Q3hS7Ixtv1Y= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: TL;DR: always call the PTE/PMD ctor, passing the appropriate mm to skip ptlock_init() if unneeded. __create_pgd_mapping() is used for creating different kinds of mappings, and may allocate page table pages if passed an allocator callback. There are currently three such cases: 1. create_pgd_mapping(), which is used to create the EFI mapping 2. arch_add_memory() 3. map_entry_trampoline() 1. uses pgd_pgtable_alloc() as allocator callback, which calls the PTE/PMD ctor, while 2. and 3. use __pgd_pgtable_alloc(), which does not. The rationale is most likely that pgtables associated with init_mm do not make use of split page table locks, and it is therefore unnecessary to initialise them by calling the ctor. 2. operates on swapper_pg_dir so the allocated pgtables are clearly associated with init_mm, this is arguably the case for 3. too (the trampoline mapping is never modified so ptlocks are anyway irrelevant). 1. corresponds to efi_mm so ptlocks need to be initialised in that case. We are now moving towards calling the ctor for all page tables, even those associated with init_mm. pagetable_{pte,pmd}_ctor() have become aware of the associated mm so that the ptlock initialisation can be skipped for init_mm. This patch therefore amends the allocator callbacks so that the PTE/PMD ctor are always called, with an appropriate mm pointer to avoid unnecessary ptlock overhead. Modifying the prototype of the allocator callbacks to take the mm and propagating that pointer all the way down would be pretty invasive. Instead: * __pgd_pgtable_alloc() (cases 2. and 3. above) is replaced with pgd_pgtable_alloc_init_mm(), resulting in the ctors being called with &init_mm. This is the main functional change in this patch; the ptlock still isn't initialised, but other ctor actions (e.g. accounting-related) are now carried out for those allocated pgtables. * pgd_pgtable_alloc() (case 1. above) is replaced with pgd_pgtable_alloc_special_mm(), resulting in the ctors being called with NULL as mm. No functional change here; NULL essentially means "not init_mm", and the ptlock is still initialised. __pgd_pgtable_alloc() is now the common implementation of those two helpers. While at it we switch it to using pagetable_alloc() like standard pgtable allocator functions and remove the comment regarding ctor calls (ctors are now always expected to be called). Signed-off-by: Kevin Brodsky --- arch/arm64/mm/mmu.c | 43 +++++++++++++++++++++++-------------------- 1 file changed, 23 insertions(+), 20 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index eca324b3a6fc..51cfc891f6a1 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -480,31 +480,22 @@ void create_kpti_ng_temp_pgd(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, int flags); #endif -static phys_addr_t __pgd_pgtable_alloc(enum pgtable_type pgtable_type) +static phys_addr_t __pgd_pgtable_alloc(struct mm_struct *mm, + enum pgtable_type pgtable_type) { /* Page is zeroed by init_clear_pgtable() so don't duplicate effort. */ - void *ptr = (void *)__get_free_page(GFP_PGTABLE_KERNEL & ~__GFP_ZERO); + struct ptdesc *ptdesc = pagetable_alloc(GFP_PGTABLE_KERNEL & ~__GFP_ZERO, 0); + phys_addr_t pa; - BUG_ON(!ptr); - return __pa(ptr); -} - -static phys_addr_t pgd_pgtable_alloc(enum pgtable_type pgtable_type) -{ - phys_addr_t pa = __pgd_pgtable_alloc(pgtable_type); - struct ptdesc *ptdesc = page_ptdesc(phys_to_page(pa)); + BUG_ON(!ptdesc); + pa = page_to_phys(ptdesc_page(ptdesc)); - /* - * Call proper page table ctor in case later we need to - * call core mm functions like apply_to_page_range() on - * this pre-allocated page table. - */ switch (pgtable_type) { case TABLE_PTE: - BUG_ON(!pagetable_pte_ctor(NULL, ptdesc)); + BUG_ON(!pagetable_pte_ctor(mm, ptdesc)); break; case TABLE_PMD: - BUG_ON(!pagetable_pmd_ctor(NULL, ptdesc)); + BUG_ON(!pagetable_pmd_ctor(mm, ptdesc)); break; default: break; @@ -513,6 +504,18 @@ static phys_addr_t pgd_pgtable_alloc(enum pgtable_type pgtable_type) return pa; } +static phys_addr_t __maybe_unused +pgd_pgtable_alloc_init_mm(enum pgtable_type pgtable_type) +{ + return __pgd_pgtable_alloc(&init_mm, pgtable_type); +} + +static phys_addr_t +pgd_pgtable_alloc_special_mm(enum pgtable_type pgtable_type) +{ + return __pgd_pgtable_alloc(NULL, pgtable_type); +} + /* * This function can only be used to modify existing table entries, * without allocating new levels of table. Note that this permits the @@ -542,7 +545,7 @@ void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; __create_pgd_mapping(mm->pgd, phys, virt, size, prot, - pgd_pgtable_alloc, flags); + pgd_pgtable_alloc_special_mm, flags); } static void update_mapping_prot(phys_addr_t phys, unsigned long virt, @@ -756,7 +759,7 @@ static int __init map_entry_trampoline(void) memset(tramp_pg_dir, 0, PGD_SIZE); __create_pgd_mapping(tramp_pg_dir, pa_start, TRAMP_VALIAS, entry_tramp_text_size(), prot, - __pgd_pgtable_alloc, NO_BLOCK_MAPPINGS); + pgd_pgtable_alloc_init_mm, NO_BLOCK_MAPPINGS); /* Map both the text and data into the kernel page table */ for (i = 0; i < DIV_ROUND_UP(entry_tramp_text_size(), PAGE_SIZE); i++) @@ -1362,7 +1365,7 @@ int arch_add_memory(int nid, u64 start, u64 size, flags |= NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start), - size, params->pgprot, __pgd_pgtable_alloc, + size, params->pgprot, pgd_pgtable_alloc_init_mm, flags); memblock_clear_nomap(start, size);