From patchwork Thu Dec 19 16:44:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 13915294 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E07E1E7718A for ; Thu, 19 Dec 2024 16:46:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E8C06B009B; Thu, 19 Dec 2024 11:46:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 76DB56B009C; Thu, 19 Dec 2024 11:46:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E7EC6B009D; Thu, 19 Dec 2024 11:46:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3C0886B009B for ; Thu, 19 Dec 2024 11:46:43 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 03E521C8C09 for ; Thu, 19 Dec 2024 16:46:42 +0000 (UTC) X-FDA: 82912286478.26.0156610 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf20.hostedemail.com (Postfix) with ESMTP id 9A8961C0010 for ; Thu, 19 Dec 2024 16:46:06 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf20.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734626776; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oGXtKyIBxO1nCcIBpzr+iCSsH+/WIbcJkid2JJ3TqdI=; b=SxzbzY4lyLV7mAAqDJSM9Yey8I2vzE/b7rXcjmfquylDcFCGiGmt1s2WLBC+rNzSY2yIVY m/+JegnS4lxTbIxKbAJE5wmzOBMO/EyCA8wLOPFKKiYQXir1E7uM6sT1CdlR3rt7NI1n5L 74DVM3IMoLqwsSXpNL2EEzmT21i7Fuc= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf20.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734626776; a=rsa-sha256; cv=none; b=XRq4LXCyu/sJ3vm3BN5ujD/A/NFGROJG/dsQ/xOzpDUN27tpFojn60MFufQHvIAvz/rEY0 Unn31KwCmHtTlCE3vBYYj/EDQPKx+Ob0q5NR2mVt8R07cKLpW52m/wfbOWN+4wwT+QL99V PvropFzNZQ39aOIhpiRPJLHr19Ssc8g= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8A3511480; Thu, 19 Dec 2024 08:47:08 -0800 (PST) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8C03F3F58B; Thu, 19 Dec 2024 08:46:36 -0800 (PST) From: Kevin Brodsky To: linux-mm@kvack.org Cc: Kevin Brodsky , Andrew Morton , Catalin Marinas , Dave Hansen , Linus Walleij , Andy Lutomirski , Peter Zijlstra , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , linux-alpha@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-kernel@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-um@lists.infradead.org, loongarch@lists.linux.dev, x86@kernel.org Subject: [PATCH 07/10] mm: Introduce ctor/dtor at P4D level Date: Thu, 19 Dec 2024 16:44:22 +0000 Message-ID: <20241219164425.2277022-8-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241219164425.2277022-1-kevin.brodsky@arm.com> References: <20241219164425.2277022-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9A8961C0010 X-Rspamd-Server: rspam12 X-Stat-Signature: 8afc9zx5a8ez51w71g76et8mkm58h61y X-Rspam-User: X-HE-Tag: 1734626766-710298 X-HE-Meta: U2FsdGVkX19+yT01I1lnPbKQO4r5c0g/v8oVCWpCZYVGGujG8aQcibCtsc18+2LnkLaHLt7nkaBTD4MEUS2wVugEVmYwF6ucs6xw55V+3si0KT5pIt/sUFV/exFzLY6eYtLmcwas59I6eYWsZzITH37KyaNmr7m9CpjXXW2ZFlPTC/JcYGx1aqKkEHwHK7e+7jVm0z9iI+Qw+U5Qu/LcBJ+203/8Q/AR2bNRbQeWhLfV8rXNZ1v9s8tjA0US8LRpgiriaVUF3QMQJ3CkKnTf/zD9HeXpxhsshw0xoWqivccGFrNaaCApe3TA73RQawwNM9pubSwAqAgbEULUDQwHs/S9o7IDrdAwA+T1VC+vQgZUpJWDtdsxC2vc5ucwnCGeClHe+lyzmv/uZ8xnCIx64AQHDsHL/d0YmM22PR9JOZQwhe+E1VZqHy64jx+ZyDJd0etVF6tTHVptNXFJlP05pVuMZgbqQ2unPWUmfSOKCVpodilZrc3ch7vN6p9hAiz2jhQEqJ5Sl3P9Dc19pov9c1m+/OTKSbT+tOO6ThUeL6YnIAqRnpdPlwyyEz0Bdra28vPYj01OUO+DcnEw5xd33fK4dWJq1ODl4JDF9g0CDD6I2BYbAPZy98KeTCwmZNscVNQosK2yr8tikvLOS4vJX/QNSf9Nbdif02dIl1vzSh1jNUYSJWHpiQS9AxkbIqjuDsV7Sm6NQIBmzWb+7y10OvLIRS/as+bAFL7GaqbrX7aRXUPgIeG81owwmb2MkTq8IeTFu2UdktdEpypMB86WoTwdQtGdCZ/siQdIVHCo1DTv1oYHqpq4xtsKqxUPP5HTD89AFFkbrlZ+7kCJ2Y6bMgDy59KZAzYoWMfrBXMhDlCpc2LR9bZwloLfKY/7en+fvygTaf4j3uSE6aCnNHM2OfQCUISunqb4QN2tWKNbtSV40/qOAP54dd/3ULbrSPgBs1fIoDyq4fqSxvNHUjH 3Buop4b5 t174rvoQS5O1o8vtThIKR+qz4cGQyRtzCTWMWSYFXw93X/28I2gzSYF1mMR5nFqSuYqg/RkdCFTCVXNuCJL4Coc1wZeGp9uS5w8Mx3TQQWJmIZP0pOyYt/06/JzaLWGZSE7cyZHzIqcnuuBwjVYyca8ioaK1kgL6lyj175z6uZXc197KPXr1ncG1k/sI4ykuxIHwqQ/O9tLcylDuIWa27MU+XZJsTsBkuCKAZ/8ioxEFM8yiEWfyeFcD4jg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Commit 55d2a0bd5ead ("mm: add statistics for PUD level pagetable") added accounting for PUD-level page tables. This patch does exactly the same thing for P4D-level page tables, introducing pagetable_p4d_[cd]tor with the same implementation as the PUD ctor/dtor and calling them on all alloc/free paths. Besides the small improvement in accounting accuracy, this also enables adding construction/destruction hooks in just one generic place for page tables at P4D level (like lower levels). Signed-off-by: Kevin Brodsky --- arch/riscv/include/asm/pgalloc.h | 8 ++++++-- arch/s390/include/asm/pgalloc.h | 12 ++++++++---- arch/x86/mm/pgtable.c | 3 +++ include/asm-generic/pgalloc.h | 2 ++ include/linux/mm.h | 10 ++++++++++ 5 files changed, 29 insertions(+), 6 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 551d614d3369..3c364ecc3100 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -108,8 +108,12 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) - riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + if (pgtable_l5_enabled) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); + riscv_tlb_remove_ptdesc(tlb, ptdesc); + } } #endif /* __PAGETABLE_PMD_FOLDED */ diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index 97db66ae06b9..85a5d07365aa 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -53,15 +53,19 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long address) { unsigned long *table = crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION2_ENTRY_EMPTY); + if (!table) + return NULL; + crst_table_init(table, _REGION2_ENTRY_EMPTY); + pagetable_p4d_ctor(virt_to_ptdesc(table)); return (p4d_t *) table; } static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!mm_p4d_folded(mm)) - crst_table_free(mm, (unsigned long *) p4d); + if (mm_p4d_folded(mm)) + return; + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + crst_table_free(mm, (unsigned long *) p4d); } static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 5745a354a241..c1bfdf7b4767 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -86,6 +86,9 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 59131629ac9c..bb482eeca0c3 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -230,6 +230,7 @@ static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long if (!ptdesc) return NULL; + pagetable_p4d_ctor(ptdesc); return ptdesc_address(ptdesc); } #define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__)) @@ -247,6 +248,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) struct ptdesc *ptdesc = virt_to_ptdesc(p4d); BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_p4d_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/include/linux/mm.h b/include/linux/mm.h index a5b482362792..e8b92f4bf3f1 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3161,6 +3161,16 @@ static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) __pagetable_dtor(ptdesc); } +static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) +{ + __pagetable_ctor(ptdesc); +} + +static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) +{ + __pagetable_dtor(ptdesc); +} + extern void __init pagecache_init(void); extern void free_initmem(void);