From patchwork Fri Jan 3 18:44:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 13925795 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57ED3E7718F for ; Fri, 3 Jan 2025 18:44:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E6CE36B009B; Fri, 3 Jan 2025 13:44:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DF56C6B009D; Fri, 3 Jan 2025 13:44:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE3626B009F; Fri, 3 Jan 2025 13:44:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B1B236B009B for ; Fri, 3 Jan 2025 13:44:57 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 78AB9C0A37 for ; Fri, 3 Jan 2025 18:44:57 +0000 (UTC) X-FDA: 82967017434.25.E804846 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf15.hostedemail.com (Postfix) with ESMTP id BDEE9A000D for ; Fri, 3 Jan 2025 18:44:55 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; spf=pass (imf15.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735929895; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bDYv+lD8IINsE762iomZ4bq5tahQltx5gkKZxMzCtrk=; b=mONYJI8RMpWhCNMyJYElV1jnY6GfiZLacy2zbw7KXFqobWlSDVEWw6VvBlEfnHNk04O58F GDhLJvstlM0QClGoZxkpuxFrs8Ky9bT1lL04/Z+xYu5gBq9PAPGAshwth9iTN+fYzCcb5r LNXFQzBOwIh+N6J+h510uPbizhoWTX8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735929895; a=rsa-sha256; cv=none; b=T/4zwqwLHBfYexJTBkaVLbNzBCi3JuEPKk6B9hF2yQJ2cub06TSMw9Kj128lZsYK8GdsOP BI3AKU8RmvdlmKDiKdYghEHK3uSPyYceS4YRp9AE8A8f6d4gjJ0Vfp3YhyjnW14M6bHj/M MAZ3PdlMojMmZtYL66sTDLOtNpifNys= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; spf=pass (imf15.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 575781C14; Fri, 3 Jan 2025 10:45:23 -0800 (PST) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2191A3F673; Fri, 3 Jan 2025 10:44:51 -0800 (PST) From: Kevin Brodsky To: linux-mm@kvack.org Cc: Kevin Brodsky , Andrew Morton , Catalin Marinas , Dave Hansen , Linus Walleij , Andy Lutomirski , Peter Zijlstra , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , Qi Zheng , linux-alpha@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-kernel@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-um@lists.infradead.org, loongarch@lists.linux.dev, x86@kernel.org Subject: [PATCH v2 6/6] mm: Introduce ctor/dtor at PGD level Date: Fri, 3 Jan 2025 18:44:15 +0000 Message-ID: <20250103184415.2744423-7-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250103184415.2744423-1-kevin.brodsky@arm.com> References: <20250103184415.2744423-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: BDEE9A000D X-Stat-Signature: ng4gg64xfsy5o6m8xsb7igwfn9fcckp1 X-Rspam-User: X-HE-Tag: 1735929895-36704 X-HE-Meta: U2FsdGVkX1+M73J/N4kWibgzGhUU3GoxEFEK6ibvH6i+hWZO7zLtpIEOzCSACy7wJqr6uecWlwU9YohsPO/2TyeI+obGBd1dV+YM91yThM2pbz6ITI/GIkEU47UUvmeBL0Z1mJCf8yLkt+9njULx7wNdveS8sRs6zMTiI2gZDwKn102SAf2I9mnu2fNTkGJ6jlD/VUa6WNNfL+3mZzyujO6/aRdsVtDx4o8jn/VUvDH2jmALR1TCjGnoE5V/zlZyAyXaSlvxpC0as36Ju1tpZ2aJNCM66yyA5Pnvjh0+M2mk/oBENodGw24n/kP6xJb0Ahg6N0EK1WBQZQr9QriZiGuOnNSm2SH0zC1qcXSHRi22ziRODom3ssR0Dj9w/M/GDWciwh+YB0SKlk4Q5DlYUct84k4G+UvtENlqV0Cu0zqWCW+lcrRi7ULx8LnwJBhYAEpVNWBAolUDyqSo1bDdo4F3Ox1/dledN+bSRv4yYXTIy5zW7HZOgGPuFkD99wXwl9dyZSqGjj7MYByL0jSGOI1hY27f43MbCRJajtNeLsKSVP/PoY1fReXtgmEdOsFYK/1W7f24yEBIdgIxMt6QUN8m6YWHHiVRRyEvUqaas4jfE5uCdbZLK9f+lP9MRxd3B8LCLvBTcJQdkRMERStDc4CGuakrhjR0RBLoN/bT2OkVoefPeNcCuvYLNiZ2GpPRLCw+rNJlctOzquqJLEmdLk3n2FrlSdMWfYOPRyBcvkicPOVSY2BeAo5CU/OqcPaDTQcJQNfIll/6YRd72ZsVV37x4WKQEmc1XIB1VjMZ2/qPQKzjtMBrF3TUoT+n0U9LWOO0bwpJdIyqhQ6q6ypOR6uDEfVyFIDWrsNNP+L/JBjz1Hw67ir1juWh8LVQK65LYxEDw9xzMYqIdPOBa9aYac4ptkkEOIZdPGASMpyFs4ewFyd0egFtuaeMUsAdq2y0srb2oJj66vWk16pJtKA FrhP1kaS La44GFHACgkKnoBXjAPwi/8O4Z7vxT0eY3ynmyvENsJPvERhGl7xmRl33ZtVCEFo3P3sbLMsOqQhw8jm6Up500hSJqS/sx5Lvpqt+xy3GoukKLn0ebMkYTjfXE7cu5e5gvDfd/S25GfyyQJOUKzn87Me18MYaLNXgRZPgXE/FjEYc0DUHRsKAx3JOyQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Following on from the introduction of P4D-level ctor/dtor, let's finish the job and introduce ctor/dtor at PGD level. The incurred improvement in page accounting is minimal - the main motivation is to create a single, generic place where construction/destruction hooks can be added for all page table pages. This patch should cover all architectures and all configurations where PGDs are one or more regular pages. This excludes any configuration where PGDs are allocated from a kmem_cache object. Acked-by: Dave Hansen Signed-off-by: Kevin Brodsky --- arch/m68k/include/asm/mcf_pgalloc.h | 3 ++- arch/m68k/mm/motorola.c | 6 ++++-- arch/s390/include/asm/pgalloc.h | 9 ++++++++- include/asm-generic/pgalloc.h | 3 ++- include/linux/mm.h | 5 +++++ 5 files changed, 21 insertions(+), 5 deletions(-) diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mcf_pgalloc.h index 22d6c1fcabfb..4c648b51e7fd 100644 --- a/arch/m68k/include/asm/mcf_pgalloc.h +++ b/arch/m68k/include/asm/mcf_pgalloc.h @@ -73,7 +73,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t pgtable) static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) { - pagetable_free(virt_to_ptdesc(pgd)); + pagetable_dtor_free(virt_to_ptdesc(pgd)); } static inline pgd_t *pgd_alloc(struct mm_struct *mm) @@ -84,6 +84,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) if (!ptdesc) return NULL; + pagetable_pgd_ctor(ptdesc); new_pgd = ptdesc_address(ptdesc); memcpy(new_pgd, swapper_pg_dir, PTRS_PER_PGD * sizeof(pgd_t)); diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index 6c09ccb72e8b..73651e093c4d 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -169,6 +169,9 @@ void *get_pointer_table(int type) case TABLE_PMD: pagetable_pmd_ctor(virt_to_ptdesc(page)); break; + case TABLE_PGD: + pagetable_pgd_ctor(virt_to_ptdesc(page)); + break; } mmu_page_ctor(page); @@ -207,8 +210,7 @@ int free_pointer_table(void *table, int type) /* all tables in page are free, free page */ list_del(dp); mmu_page_dtor((void *)page); - if (type == TABLE_PTE || type == TABLE_PMD) - pagetable_dtor(virt_to_ptdesc((void *)page)); + pagetable_dtor(virt_to_ptdesc((void *)page)); free_page (page); return 1; } else if (ptable_list[type].next != dp) { diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index 5fced6d3c36b..b19b6ed2ab53 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -130,11 +130,18 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) static inline pgd_t *pgd_alloc(struct mm_struct *mm) { - return (pgd_t *) crst_table_alloc(mm); + unsigned long *table = crst_table_alloc(mm); + + if (!table) + return NULL; + pagetable_pgd_ctor(virt_to_ptdesc(table)); + + return (pgd_t *) table; } static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) { + pagetable_dtor(virt_to_ptdesc(pgd)); crst_table_free(mm, (unsigned long *) pgd); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index de4df14158e6..892ece4558a2 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -271,6 +271,7 @@ static inline pgd_t *__pgd_alloc_noprof(struct mm_struct *mm, unsigned int order if (!ptdesc) return NULL; + pagetable_pgd_ctor(ptdesc); return ptdesc_address(ptdesc); } #define __pgd_alloc(...) alloc_hooks(__pgd_alloc_noprof(__VA_ARGS__)) @@ -280,7 +281,7 @@ static inline void __pgd_free(struct mm_struct *mm, pgd_t *pgd) struct ptdesc *ptdesc = virt_to_ptdesc(pgd); BUG_ON((unsigned long)pgd & (PAGE_SIZE-1)); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); } #ifndef __HAVE_ARCH_PGD_FREE diff --git a/include/linux/mm.h b/include/linux/mm.h index 065fa9449d03..486638d22fc6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3243,6 +3243,11 @@ static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) __pagetable_ctor(ptdesc); } +static inline void pagetable_pgd_ctor(struct ptdesc *ptdesc) +{ + __pagetable_ctor(ptdesc); +} + extern void __init pagecache_init(void); extern void free_initmem(void);