From patchwork Thu Dec 19 16:44:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 13915288 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C01E9E7718A for ; Thu, 19 Dec 2024 16:46:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 54CB86B0082; Thu, 19 Dec 2024 11:46:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4AEC46B0083; Thu, 19 Dec 2024 11:46:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 300966B0085; Thu, 19 Dec 2024 11:46:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 112426B0082 for ; Thu, 19 Dec 2024 11:46:18 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C277BB0E7B for ; Thu, 19 Dec 2024 16:46:17 +0000 (UTC) X-FDA: 82912284798.03.6472C7B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf21.hostedemail.com (Postfix) with ESMTP id 73A581C0002 for ; Thu, 19 Dec 2024 16:45:12 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf21.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734626761; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vxpu9hpxVezNNfPXO+EkxJEzB3KeAziYK7om5MQJ0nQ=; b=G2k+iTPyQGl5eHV8IHbRbzpaRqpB+Ja9LA+ZJ2fA7d3pP09mwmNLbZu9thC4x80ybNLZyd 3Y2EDse70/63vzMTpsefJ/QfJGMXynHVUWH3FkmKur9PAnev0UjDfa79WC26Qw+ccvxbbl Nsq5unpXwCZOaF+Beuksj+1BFBmsF2c= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf21.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734626761; a=rsa-sha256; cv=none; b=e6wUNBii25aW2V1uWHEWLH/O4HSG53UkMQuV5tWnUh3ulUasw+ANJCgwlGRRlOEN75GKXb 75qAUbO7uLBM5XPmJozQLubutimCkwx/UYZ0i+pFQO1mgrA7EvALl5sqkiNTKMlhrD7Ovx WEm9bUocuF9iYFiPJ/i8WdNPhXRGkxI= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5FF7A1D13; Thu, 19 Dec 2024 08:46:43 -0800 (PST) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 60DB33F58B; Thu, 19 Dec 2024 08:46:11 -0800 (PST) From: Kevin Brodsky To: linux-mm@kvack.org Cc: Kevin Brodsky , Andrew Morton , Catalin Marinas , Dave Hansen , Linus Walleij , Andy Lutomirski , Peter Zijlstra , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , linux-alpha@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-kernel@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-um@lists.infradead.org, loongarch@lists.linux.dev, x86@kernel.org Subject: [PATCH 01/10] mm: Move common parts of pagetable_*_[cd]tor to helpers Date: Thu, 19 Dec 2024 16:44:16 +0000 Message-ID: <20241219164425.2277022-2-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241219164425.2277022-1-kevin.brodsky@arm.com> References: <20241219164425.2277022-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 73A581C0002 X-Stat-Signature: 8b9ff195ghtnigaf4qgqcx3kumo98ixm X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1734626712-250855 X-HE-Meta: U2FsdGVkX19hG6A14f+L4OUEeOkPl9n5LjZHtqBjtTbuaxsfeTTUokXdnzykzOXIG4oZj1v3AZSUCcZ/NHJRyxkYbL/RFi52TfGjx4ccSvosNS+1XK4gftaNlac+1LMIGZQWfJe6D0U8KpmcuDl8W6J7trjPGkQur58q+NxiibwKOE/M2RUTwfUyF/nGhNBaZxtgstvt5xyp5f8Ii29+NllDCItrUL4i3uzloyZQ6rqIB1Iul5wqjRxyZ0re4D2UnNwJas2UffD+voqC8m4/PnPWgTPNoLw0j5VxdkMRNj5rOGGa1kvTEC3E7jhJs/C3IPKkC6hCnaRCyWgPOS48G7D5RA74qSyflkT+j5gkkO4DINreELcU070fT3WFAk9FyAJae/DGj5zMsZtiLhI9Wic4UXMo2ySxT1YW47Ny3th6CD2PbmEG2IfEsMdb86wumjBPhfXxXx1xnUEyyCRqL+9lV5yJqhgmX9ivaF7OApWPAlYBftP06h+R3efrKuSEgsAxqviYp7UJblbYfwn1A5TnT88dsOh+4aeirgkWaoyGyAKkerzp9PqxBelwZmyk6pTUuewPmUU8mVGGI8XO35VyH/yEeqGqyh1nY6V1YX1D79qjT2BWzDToKmYgIKObC25ZW3a+4Ld8zsz1NrZfZtIfiRVClClw/Npqa6A7lgXVd4n19C7xDajCzrjqTufAarKLjl7ZBaHCLf8K/fzusFmvbmtHvIeXN4rg3tQeU2m4syh3/XdBFnoZ4n51zKXtXH7/cFE0HGJ4hk1cgGKYgwiL2jXJUaQDplZPq2XpaXugVmYm02WiVkwFsv1mAyNNI7CxfGKH74iAQgghOXB5iEvaRtDNqIonpCRkAbtorSjx8TzH4wuuOS3geq78uC5oR48ebbKjYHUYxpqFSKZKCuNVHEuYrgryBpEc+N8MGE0FzA1vXYk/+y9PSQNzJVbE6nZB8iPbSQbq47OE2Cc nu2IXB5w Q+y5XOuwKOxlwY0/L1rctn2sN17+Yg7WN7ZuBS2wY7pNsbE0RczkIlyEpwgKwpmPJQGx44FdOXij6PfGZ8OI7rBIaobAqKonGTWljnBjQ5ou4HrwlqPy9I7xQuWHZyIFzRYCM332zRqAMLbM+LfOxy7bn94qQgDGCCWaa4eFqRreTjGhMkBXBnCIFpJ35pC8nS+tvonuVe+K3Fi+SiE+fXg3Ee3jhFiTIyIMJC80mMRWfU/TKbIV0V9RK4LyCRhvJaMsFhOkeZ/MLdiXJwedA4LbTu/iR24JyOTdSbc5NLsTdaDDz2h0TEc7QXY4Iw9gnDVlIltaeqp0ks1Y= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Besides the ptlock management at PTE/PMD level, all the pagetable_*_[cd]tor have the same implementation. Introduce common helpers for all levels to reduce the duplication. Signed-off-by: Kevin Brodsky --- include/linux/mm.h | 46 ++++++++++++++++++++++------------------------ 1 file changed, 22 insertions(+), 24 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index c39c4945946c..a5b482362792 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2915,6 +2915,22 @@ static inline void pagetable_free(struct ptdesc *pt) __free_pages(page, compound_order(page)); } +static inline void __pagetable_ctor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_set_pgtable(folio); + lruvec_stat_add_folio(folio, NR_PAGETABLE); +} + +static inline void __pagetable_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + #if defined(CONFIG_SPLIT_PTE_PTLOCKS) #if ALLOC_SPLIT_PTLOCKS void __init ptlock_cache_init(void); @@ -2992,22 +3008,16 @@ static inline void ptlock_free(struct ptdesc *ptdesc) {} static inline bool pagetable_pte_ctor(struct ptdesc *ptdesc) { - struct folio *folio = ptdesc_folio(ptdesc); - if (!ptlock_init(ptdesc)) return false; - __folio_set_pgtable(folio); - lruvec_stat_add_folio(folio, NR_PAGETABLE); + __pagetable_ctor(ptdesc); return true; } static inline void pagetable_pte_dtor(struct ptdesc *ptdesc) { - struct folio *folio = ptdesc_folio(ptdesc); - ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); + __pagetable_dtor(ptdesc); } pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp); @@ -3110,22 +3120,16 @@ static inline spinlock_t *pmd_lock(struct mm_struct *mm, pmd_t *pmd) static inline bool pagetable_pmd_ctor(struct ptdesc *ptdesc) { - struct folio *folio = ptdesc_folio(ptdesc); - if (!pmd_ptlock_init(ptdesc)) return false; - __folio_set_pgtable(folio); - lruvec_stat_add_folio(folio, NR_PAGETABLE); + __pagetable_ctor(ptdesc); return true; } static inline void pagetable_pmd_dtor(struct ptdesc *ptdesc) { - struct folio *folio = ptdesc_folio(ptdesc); - pmd_ptlock_free(ptdesc); - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); + __pagetable_dtor(ptdesc); } /* @@ -3149,18 +3153,12 @@ static inline spinlock_t *pud_lock(struct mm_struct *mm, pud_t *pud) static inline void pagetable_pud_ctor(struct ptdesc *ptdesc) { - struct folio *folio = ptdesc_folio(ptdesc); - - __folio_set_pgtable(folio); - lruvec_stat_add_folio(folio, NR_PAGETABLE); + __pagetable_ctor(ptdesc); } static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) { - struct folio *folio = ptdesc_folio(ptdesc); - - __folio_clear_pgtable(folio); - lruvec_stat_sub_folio(folio, NR_PAGETABLE); + __pagetable_dtor(ptdesc); } extern void __init pagecache_init(void); From patchwork Thu Dec 19 16:44:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 13915289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA422E77184 for ; Thu, 19 Dec 2024 16:46:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D3AB6B0085; Thu, 19 Dec 2024 11:46:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 65BE16B008C; Thu, 19 Dec 2024 11:46:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4AE866B0092; Thu, 19 Dec 2024 11:46:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 28B4A6B0085 for ; Thu, 19 Dec 2024 11:46:22 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C436D1C8A41 for ; Thu, 19 Dec 2024 16:46:21 +0000 (UTC) X-FDA: 82912285932.11.872A44B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf02.hostedemail.com (Postfix) with ESMTP id AB5028000D for ; Thu, 19 Dec 2024 16:45:16 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf02.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734626765; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PwcZYQebHQhJI8S2XCpqJM+lG6xJt9XpWgsLOxw4To8=; b=A6SDRSNRGLSH122FDaEqe4xe6DSEurwOPclC4Bm6joizRb8DoacmFfiRtF6YsI/OJ1hYWf s5+bwje+V8MIkz8AQ5SUjmnl/ZUgsj1r64W6Qe4xsiclApnOtRJMtzyy8n2yG7QLCN9KeA IeRazwz1aMysBQBK6lRLECEx7reTMj0= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf02.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734626765; a=rsa-sha256; cv=none; b=jU8jvF/ArRTIHY+KsIVMA3hDPIR7jJwRsQgSpRWWv4NHRfZfAv4lJQu7ysoPRgbkbLUyaw D/ZdOKqp0HearZJ8LwaoScGsBm6rjk69okEFAvyXZqRGsMqbk98vsUdLHooTyZr4ihBJ9T SiyXGeYf8QhrDzqotRPGJ/jqJnyamao= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 90DDF1480; Thu, 19 Dec 2024 08:46:47 -0800 (PST) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 90EAC3F58B; Thu, 19 Dec 2024 08:46:15 -0800 (PST) From: Kevin Brodsky To: linux-mm@kvack.org Cc: Kevin Brodsky , Andrew Morton , Catalin Marinas , Dave Hansen , Linus Walleij , Andy Lutomirski , Peter Zijlstra , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , linux-alpha@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-kernel@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-um@lists.infradead.org, loongarch@lists.linux.dev, x86@kernel.org Subject: [PATCH 02/10] parisc: mm: Ensure pagetable_pmd_[cd]tor are called Date: Thu, 19 Dec 2024 16:44:17 +0000 Message-ID: <20241219164425.2277022-3-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241219164425.2277022-1-kevin.brodsky@arm.com> References: <20241219164425.2277022-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: AB5028000D X-Stat-Signature: 74rtipxxiwnqwqzrccf99qzqga49k47y X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1734626716-584929 X-HE-Meta: U2FsdGVkX18jQruJWpBqauFV6fPhdvWjMgTrQwEadP7kvFQvCiTSEb01hMxMKA2F/BLc3MDeCb/nob9BNC+EGHaRc8WOJtqMoWmz6ngDE5q29IMMy2QpYU+c6dSjvjcvGJx9nnwZuOUvI8nkObGViKN11NHpZCvook208Z2fnBSxx/9w803JPORdZxSS1PxOStF2zSf5swtrAzhFd2Ds9JWtuK4Qpk+9N2srWdRmV6ikQiyo9CbOa5ATqKzq9NyjbX5WofR+4l2pduIQl9CcChqae5RHE5pgakRX/F6HJvvJp2OBIGPAoZ/dnRYV3E9AVGiFHJDHGHTnRxtgc9cKDY7pxksLHaeRtHcfh0Tl6vaV4q9H5Wn2VKdsTsi0EU3ahd59aIy9Opzk7oGIEQ4rGwNucxnQBKvLZPY1mIYEWEYqacMzcchHuAx6IE40gsFwkDBigW0Fqb6NkNhwypUjeTYQJEuQitmyS2ihcfL8LHg+UVIMBHw/4cfcs8r9yjomOFGtUF75hOtw1XWPxB0At/xt9Ndj6qVszMXl9yU5ls2J0RHTM6R548rhltxfTYYjsGl+v7GR1l75osSuYkg37TyhWIB5yXBqifxJj/xadP+Z7zPg0iko6awkBkOZac3JlcIO4N0kZTTnpAv+MkLM51T/xZUDZt9j9iWasD6cX5O5LKLzYZ6iSjPcgnyV36FKboy6YtWQfY/XPrEulo7TiheaRg88GBXdVvv2krkhaRdU9a7oXpZw8CNtfeZ7pSZuX2ZBE8LGwySUf4mQErOLVxFiP548NO/bbCNhusHhSxRtqeEHxheatwZJ+m2PQT9VjOj2SSKBjz3Lh3aZnJBr5dlCDECh8M4rgJyEezJ7kzqdeRXzNMeq+EID+Ysqx/3WSGplg7YovmRIsjfauQ8QY0CpgaacPKTd7WA9s/kAChJqCtSpmsjnXST/86qSOAyAJBlN39PGxg0z97YNw/M bGYQJuGJ jPDtu28Mu2RVwW2Nfb3SxEXBN7CpgslNZbGEzPBgyjmUmoWSxH2qpvyLEbqqYMfMHmo0qWBvTHgbE4qS3zwXKVYCwk+WrXbasoQOnRSVKfWXIBKp9a+x2EWN+/24MMH+Jdu84XUJcdkciInNac25d29MmA1zSsQSsczcK1T4DqEKHPAvY6bG7tK03uou+OBCHLbcLZpGKFOT5iFdEuyvIATKtU3Kk7IN7aBlfu17Tubvw/3+V//BJNlPPUDfS6qWlLUdBlBKyEpMjtArcE2djkxKj6DDjvQ2biPgP4IZWfK3p5qYPqHiuEsCG/+rVlPQUva5Q X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The implementation of pmd_{alloc_one,free} on parisc requires a non-zero allocation order, but is completely standard aside from that. Let's reuse the generic implementation of pmd_alloc_one(). Explicit zeroing is not needed as GFP_PGTABLE_KERNEL includes __GFP_ZERO. The generic pmd_free() can handle higher allocation orders so we don't need to define our own. These changes ensure that pagetable_pmd_[cd]tor are called, improving the accounting of page table pages. Signed-off-by: Kevin Brodsky --- arch/parisc/include/asm/pgalloc.h | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/arch/parisc/include/asm/pgalloc.h b/arch/parisc/include/asm/pgalloc.h index e3e142b1c5c5..3e8dbd79670b 100644 --- a/arch/parisc/include/asm/pgalloc.h +++ b/arch/parisc/include/asm/pgalloc.h @@ -11,7 +11,6 @@ #include #define __HAVE_ARCH_PMD_ALLOC_ONE -#define __HAVE_ARCH_PMD_FREE #define __HAVE_ARCH_PGD_FREE #include @@ -46,17 +45,19 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address) { - pmd_t *pmd; + struct ptdesc *ptdesc; + gfp_t gfp = GFP_PGTABLE_USER; - pmd = (pmd_t *)__get_free_pages(GFP_PGTABLE_KERNEL, PMD_TABLE_ORDER); - if (likely(pmd)) - memset ((void *)pmd, 0, PAGE_SIZE << PMD_TABLE_ORDER); - return pmd; -} - -static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) -{ - free_pages((unsigned long)pmd, PMD_TABLE_ORDER); + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + ptdesc = pagetable_alloc(gfp, PMD_TABLE_ORDER); + if (!ptdesc) + return NULL; + if (!pagetable_pmd_ctor(ptdesc)) { + pagetable_free(ptdesc); + return NULL; + } + return ptdesc_address(ptdesc); } #endif From patchwork Thu Dec 19 16:44:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 13915290 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77FC4E77184 for ; Thu, 19 Dec 2024 16:46:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0B9056B0092; Thu, 19 Dec 2024 11:46:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 041466B0093; Thu, 19 Dec 2024 11:46:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DFD296B0095; Thu, 19 Dec 2024 11:46:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BEBF76B0092 for ; Thu, 19 Dec 2024 11:46:26 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 75B92C175D for ; Thu, 19 Dec 2024 16:46:26 +0000 (UTC) X-FDA: 82912285806.12.EA86838 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf12.hostedemail.com (Postfix) with ESMTP id 20F4440020 for ; Thu, 19 Dec 2024 16:46:09 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf12.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734626753; a=rsa-sha256; cv=none; b=FXXNrQcuDHd9G/RejwReO8QtoNR+xkVTph0RvXY6gd0Hf7HLhLAqoDwqg5qaCqzaDeT4l2 ymnXfsZM/+rGWbIvrVy5FOYC2FM4g8NCThyfgmDYLoJ1xO0DB9d2QvB+9qkngn867MV8Ne sXDVJmgVZObOfwupFz5KCVbFvoHlG7E= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf12.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734626753; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=URvkOy53UcBsPFjkdm3DY6eWupuu8tvChY7EKfoY7IQ=; b=MNGnidrBkD+vQ+WQwFyTcrJQ5U9QWbczAYNaVgrbVNXZmuntAgca9HY2vZdmb4Vx1aQveQ LSzS5pu+O3efFPStS7qgCCuf4NCTrIlfFv/Y4Jwo5uoHBM32Unb+58SboLrwHi1hPw+BiN yuF3Vt4I4lqMNQINY80b1x6s48ZXxgs= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BEB531480; Thu, 19 Dec 2024 08:46:51 -0800 (PST) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C069C3F58B; Thu, 19 Dec 2024 08:46:19 -0800 (PST) From: Kevin Brodsky To: linux-mm@kvack.org Cc: Kevin Brodsky , Andrew Morton , Catalin Marinas , Dave Hansen , Linus Walleij , Andy Lutomirski , Peter Zijlstra , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , linux-alpha@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-kernel@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-um@lists.infradead.org, loongarch@lists.linux.dev, x86@kernel.org Subject: [PATCH 03/10] m68k: mm: Add calls to pagetable_pmd_[cd]tor Date: Thu, 19 Dec 2024 16:44:18 +0000 Message-ID: <20241219164425.2277022-4-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241219164425.2277022-1-kevin.brodsky@arm.com> References: <20241219164425.2277022-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-Stat-Signature: kks65d5ck11gq5bdgc9zz4cs7x85rxo5 X-Rspam-User: X-Rspamd-Queue-Id: 20F4440020 X-Rspamd-Server: rspam08 X-HE-Tag: 1734626769-992806 X-HE-Meta: U2FsdGVkX1/D5+hxxSzab61p/01BCUh/wt9ZNDsC5/uwekMC5Kj+2ASDCktcIhWsQJddQ8Hps6MR3NvhH+3n7QE4fyyNVEF2f31Ct+lEpU6FU3D5aQBKJ3YxfkzvounQamQfq87wbMQNnVMoeoDJsPtt/Eocumimce02D/QveKgUvm/ZlfiYjeU0mY6jJg5jPZ+ItLTlirM7+/5YVpC0FU4KPhbRwyLVgToRD7e46gOTl+/grkBxNeCzimX+uPgx4axmQQnGjfQvS9l/Ikyezkui/coNuW7vj64dSggXR+P/ZC3LHUx28ZghDZQd/k70tgZmIiwKQj5hnY5Ziht7xt5rQrVjO3+FA6ANxY/xIC2927Vr0eNdaXCMVSYTMrJDCYEJ3k3dmMRSZECh3ys41+JzAbTBjoNTQIGVEl5cNxDT6a0h9PkBvDgFhKyRX1RWZcl8dvVQ4zkK3goIYAr/YS7T+p3l4pnsVXPtTk9JqaCYm1wTBreJ4M9Z+uwieVxhzx5wWqrI8kFCifsnCkurCQrtRpd72DlZEmCZ4OBN+R2Uyu3T3wNZFhsMOrc/S+qEquppjUlEhCPbXjZhS0xsu7UKavfk670xsLFV1yiGdUDCLi64fL1F99TACSj1dRAMR3348rdoZ1cx+L08M2mIlL3Gi0iuedFviDWqaAVHx1aSukKwNIqneRvseU55gsHOKF1SosQ1Xxvp739HdMexMIBPQZSRt7IzAtz+FFP1mkC11dcgz5hZW/zp7T80A5f7Y19UllpsrB37jKdsxM3kyNpJyIywggBpd217a3pTfBIOmxtmtO0KfWWz9lS6BwyHnpvmygkb1Sjq6xV/avc2aLaPQ3rjJvqezzwyYOSyAQHJXdG4kTD6TgiEcDOH86awxunRvxdubvgoeBWh3/Vnl8NZqx8a0YMg+WtY/FIOaP32NBuVGWXL/ScNJglQAR2F+GxQXyUIU2/gJ74xn21 YqHO1GaY BFnhphRBzuoz6AY8/CkMatjf8QpCn3822sz3fEqDb/bntefN1gssvHCU1Uj3cF969iwaG7bBeVFqgYD9uej8szWvYhwUdTNFmy7sMlVYCtaHPkr85O7fZol3+tFhOtIkH1sQ+PevdRO8n+W76QcchsePk+9owbQogAmSJNFROUUnM7IzmI/gyIj8fX+JpdDMrghJB15x2mu82cKr5RyCkhpbf4TKOYIdDAeOIy762BQWJuQt0oJt2BWmEwnt1U8w/SirIqdf7ezCLHS5j6G+YS0/7ZxNMpYyuxzOZp/+6cSRnQnpwUC9XDuZ/no/+9avoleGqM/yHmepdfs4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: get_pointer_table() and free_pointer_table() already special-case TABLE_PTE to call pagetable_pte_[cd]tor. Let's do the same at PMD level to improve accounting further. TABLE_PGD and TABLE_PMD are currently defined to the same value, so we first need to separate them. That also implies separating ptable_list for PMD/PGD levels. Signed-off-by: Kevin Brodsky --- arch/m68k/include/asm/motorola_pgalloc.h | 6 +++--- arch/m68k/mm/motorola.c | 25 +++++++++++++++++++----- 2 files changed, 23 insertions(+), 8 deletions(-) diff --git a/arch/m68k/include/asm/motorola_pgalloc.h b/arch/m68k/include/asm/motorola_pgalloc.h index 74a817d9387f..5abe7da8ac5a 100644 --- a/arch/m68k/include/asm/motorola_pgalloc.h +++ b/arch/m68k/include/asm/motorola_pgalloc.h @@ -9,9 +9,9 @@ extern void mmu_page_ctor(void *page); extern void mmu_page_dtor(void *page); enum m68k_table_types { - TABLE_PGD = 0, - TABLE_PMD = 0, /* same size as PGD */ - TABLE_PTE = 1, + TABLE_PGD, + TABLE_PMD, + TABLE_PTE, }; extern void init_pointer_table(void *table, int type); diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index c1761d309fc6..37010ee15928 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -97,17 +97,19 @@ void mmu_page_dtor(void *page) typedef struct list_head ptable_desc; -static struct list_head ptable_list[2] = { +static struct list_head ptable_list[3] = { LIST_HEAD_INIT(ptable_list[0]), LIST_HEAD_INIT(ptable_list[1]), + LIST_HEAD_INIT(ptable_list[2]), }; #define PD_PTABLE(page) ((ptable_desc *)&(virt_to_page((void *)(page))->lru)) #define PD_PAGE(ptable) (list_entry(ptable, struct page, lru)) #define PD_MARKBITS(dp) (*(unsigned int *)&PD_PAGE(dp)->index) -static const int ptable_shift[2] = { - 7+2, /* PGD, PMD */ +static const int ptable_shift[3] = { + 7+2, /* PGD */ + 7+2, /* PMD */ 6+2, /* PTE */ }; @@ -156,12 +158,17 @@ void *get_pointer_table(int type) if (!(page = (void *)get_zeroed_page(GFP_KERNEL))) return NULL; - if (type == TABLE_PTE) { + switch (type) { + case TABLE_PTE: /* * m68k doesn't have SPLIT_PTE_PTLOCKS for not having * SMP. */ pagetable_pte_ctor(virt_to_ptdesc(page)); + break; + case TABLE_PMD: + pagetable_pmd_ctor(virt_to_ptdesc(page)); + break; } mmu_page_ctor(page); @@ -200,8 +207,16 @@ int free_pointer_table(void *table, int type) /* all tables in page are free, free page */ list_del(dp); mmu_page_dtor((void *)page); - if (type == TABLE_PTE) + + switch (type) { + case TABLE_PTE: pagetable_pte_dtor(virt_to_ptdesc((void *)page)); + break; + case TABLE_PMD: + pagetable_pmd_dtor(virt_to_ptdesc((void *)page)); + break; + } + free_page (page); return 1; } else if (ptable_list[type].next != dp) { From patchwork Thu Dec 19 16:44:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 13915291 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 282DDE77184 for ; Thu, 19 Dec 2024 16:46:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B40836B0095; Thu, 19 Dec 2024 11:46:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AC8C46B0096; Thu, 19 Dec 2024 11:46:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 942886B0098; Thu, 19 Dec 2024 11:46:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 74DB66B0095 for ; Thu, 19 Dec 2024 11:46:30 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 33764A1802 for ; Thu, 19 Dec 2024 16:46:30 +0000 (UTC) X-FDA: 82912285638.13.2150C42 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf23.hostedemail.com (Postfix) with ESMTP id 77583140006 for ; Thu, 19 Dec 2024 16:46:06 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf23.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734626773; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3WEDzI0flGVpS0CVH2Q86E+EzbPPJ+nig/lm6WiqDI4=; b=ejbvsrOpufykBdhdg+8NW/TTxmGGH5mnuv21eX8DddvqrLR3Lqaz3E2fAIdFQMuyRNTZUz njXHIlZ8VQoXQ+ITbnyyAidlrA0qhVeLAjsDvh4uo8IhoMCgdl+DGeb3XIuUO+S0z1jZTv Nti5ztMi+jX9XVipMZAOakgmVW/TrQE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734626773; a=rsa-sha256; cv=none; b=pY1n1HkX05fvXphin1nlfWA1xv+h2/3RxkDT95ojmpgjaAMtRLiy0PWwC7VWJAh73/SMVJ FLrc936SbghrrVo4JxgbrTTIgesLdO7sXIAr9jjzBcxGa97KNa3uMKFMxH69l53D31XDt5 lxhasze1zVJsWcaz+UtDB3MqYrGUADE= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf23.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EE9F01480; Thu, 19 Dec 2024 08:46:55 -0800 (PST) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id F018D3F58B; Thu, 19 Dec 2024 08:46:23 -0800 (PST) From: Kevin Brodsky To: linux-mm@kvack.org Cc: Kevin Brodsky , Andrew Morton , Catalin Marinas , Dave Hansen , Linus Walleij , Andy Lutomirski , Peter Zijlstra , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , linux-alpha@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-kernel@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-um@lists.infradead.org, loongarch@lists.linux.dev, x86@kernel.org Subject: [PATCH 04/10] s390/mm: Add calls to pagetable_pud_[cd]tor Date: Thu, 19 Dec 2024 16:44:19 +0000 Message-ID: <20241219164425.2277022-5-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241219164425.2277022-1-kevin.brodsky@arm.com> References: <20241219164425.2277022-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-Stat-Signature: d1jcwyw6mhxfaumqi1pq8ysnif6qpbzj X-Rspamd-Queue-Id: 77583140006 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1734626766-393545 X-HE-Meta: U2FsdGVkX1/Kg5DG+nYyKhk0M7pawsr7Zi+T836lAa1NsHQABRUm3q6pPPsFPWUzQcAH0VkDknaFFXfdVOD8TEsg87RDzOU1P2DHviWdknfWRHWwYak/BKjiiahuSXw8iGZy1WOxaEME51BjRLwSM3A3qyXtwU0x34ZQkBBK4gHL2C0Os1T8FAu2nJOBcW3EF8VQBnqyfJlDTkP0X81WmBeEyGPBT8qSli/jEi13fe3J4+w3vtLD+I6H1igcSnxPlRqR+/NPTKsMyPLJA5bJliVE65jWKy42Eun64XaHSHoEMkFUy+xnulQEBE7Z9VrD5scwnEDldxH10T1VPKPD+dGzmabJNNsUa/6JQYGkCPImRO+6e1vtog+U4ZQVfceQtK//hry7Hapa6XwyVd8F8KMawf1Ktooa1Paj8NGkwtxdAhktQcRuH1p9hrTIXlI22xHQupPboJS068aW8CM8kRKiZXzmPvKkwA15Y3HMlbrdFysqWFhtgwaPlOlH9q7MCNMt+a3K3n6fiVzJWPIqrQqHADoW1RweWBpMXBtYF32PaF6Zh4ldWe61jv75PkjsMUlJcEWBdVOMjmRHekvfxSh1jTkAuLPH+bg9D2LK4nHvlH+4wt+V+lfAIlzAwplf2dyxXYV/aAuuVt+rDTpIh0TLohlfx671qTge+zY1eyuYVQbB4gBIMjbXRajN0aia9F5u/pgefF2LVix8ONA3KQrqIckzA1DDAFmQG4Vg9AYB61TipPjTgTJmej2um8EbbgRCq9dfd64jCoHdYA5/LwYb6aKkappggZb5DhOVyAwF0/IGgHrF2Eiagz4dILh4YFzouSZmVaLm/VOC5UFlI+KlhpdeNNld2W61BISkHVYEaZtpTvlGN6WgoeQ1nqQhamc8DE+ynDEazxP72YrXjLUaKKCGNteR0mRCSP7sqZ+rccQjJVMKmgbNG1eFYrH6a/UAQhH959u02D+eQtU j/gEEsxM So9P9m8CslbVY4fMzUEaHqy+76TF9gCGqxLA9AEmRTpDmlUHZivJO9G+86hx5/whY5FLrCWbndf38DFQVlwFSUCzeGY5XoQ3sREJgwxL+TQ42QpcP3CdA7MK5+2pfKcupTrt7+OaZF0YoIzAKL+KHvJHo1Sy/ApgxvJP53NIQoadO0KvnvK4uTh0Ads70xk0Tkf5Lae5YnYL2DJcRzapEOSdsqkn/sDxVjiZ6Y4fifAQCRc76+P3BM/Qn9L9Nlcsj1rPZNnCw0CArRF61kUL/6krx5rFHvTjOiFVTgUqQmPNhvPGotzs+6BRlLbifvKpat8i0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Commit 55d2a0bd5ead ("mm: add statistics for PUD level pagetable") introduced PUD-level ctor/dtor helpers to improve the accounting of page table pages. s390 doesn't use the generic pgalloc implementation and it seems that it got missed in the process. Add the missing calls to the ctor/dtor helpers in pud_alloc_one/pud_free to match the other architectures. Signed-off-by: Kevin Brodsky --- arch/s390/include/asm/pgalloc.h | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index 7b84ef6dc4b6..97db66ae06b9 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -67,15 +67,20 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) { unsigned long *table = crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION3_ENTRY_EMPTY); + + if (!table) + return NULL; + crst_table_init(table, _REGION3_ENTRY_EMPTY); + pagetable_pud_ctor(virt_to_ptdesc(table)); return (pud_t *) table; } static inline void pud_free(struct mm_struct *mm, pud_t *pud) { - if (!mm_pud_folded(mm)) - crst_table_free(mm, (unsigned long *) pud); + if (mm_pud_folded(mm)) + return; + pagetable_pud_dtor(virt_to_ptdesc(pud)); + crst_table_free(mm, (unsigned long *) pud); } static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long vmaddr) From patchwork Thu Dec 19 16:44:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 13915292 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CFA5E77184 for ; Thu, 19 Dec 2024 16:46:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DFEF86B007B; Thu, 19 Dec 2024 11:46:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D87B36B0098; Thu, 19 Dec 2024 11:46:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C29016B0099; Thu, 19 Dec 2024 11:46:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A27BF6B007B for ; Thu, 19 Dec 2024 11:46:34 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 550D5A17DD for ; Thu, 19 Dec 2024 16:46:34 +0000 (UTC) X-FDA: 82912286814.14.7A5EF0B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf10.hostedemail.com (Postfix) with ESMTP id 453F4C0014 for ; Thu, 19 Dec 2024 16:46:18 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf10.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734626777; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lZA9kLthfPyIxwHp01Yxv5Ht/fhkFKG3TTGvCJtNyWQ=; b=TLORXml9Dl6Kh9L7qN+S2jKCLJIWQfY9vXpB8xtnoC1FUhgeazziUaxUn/AuHL+btO8A8x lqEhInV2N8M+2/kU965AmwIp4JRqzndCTqRfTCpgHc8G+dLE+QRimusV2eP8Htj19EZSwV 0H4idNIR4Eo+Eb1PGDqfvFaLO62++y0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734626777; a=rsa-sha256; cv=none; b=EblwEsIRGJ20MlMbQ1wCeQtLmxwhFBudz5TVurZkPbkJBPNAhfKS4nEqUdavRRrfvFb5cf eZj3UVPpWUZ271o89ZFXqudsRcy+8vfZ0dVaeJua0cBsaN3k7UePgKLJFvZhALwSVJXjCZ 7xyS2ACmGU5XSxgqGt7CUTOR4oDWKGA= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf10.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2A70A1480; Thu, 19 Dec 2024 08:47:00 -0800 (PST) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2B9613F58B; Thu, 19 Dec 2024 08:46:28 -0800 (PST) From: Kevin Brodsky To: linux-mm@kvack.org Cc: Kevin Brodsky , Andrew Morton , Catalin Marinas , Dave Hansen , Linus Walleij , Andy Lutomirski , Peter Zijlstra , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , linux-alpha@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-kernel@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-um@lists.infradead.org, loongarch@lists.linux.dev, x86@kernel.org Subject: [PATCH 05/10] riscv: mm: Skip pgtable level check in {pud,p4d}_alloc_one Date: Thu, 19 Dec 2024 16:44:20 +0000 Message-ID: <20241219164425.2277022-6-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241219164425.2277022-1-kevin.brodsky@arm.com> References: <20241219164425.2277022-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-Stat-Signature: azr88tpu46u4szhpfn44zi7rgjzct3pm X-Rspamd-Queue-Id: 453F4C0014 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1734626778-299104 X-HE-Meta: U2FsdGVkX1/Wx9Sx8hZtow3XMRhnH/uAbBEuFeNOaRkY2drkTYsY1G1K5hGfv3NdieyOVxS81pz/6JwadeErmWcmPrrtbtLIj5I9A9hTjESMRPw47xlT1yH9oRlk00lz5ZcdtXG8NJ/ybKI25j257Vj5EWX78UPPDXEa6l7L0mA1bFDNC3H9BTnNQLNc1HGw839DJ2IHIYYDUJK3KH6uWNtIMEI1NC7Y85HoXHhbpd8pHL6YNTJAq8tqWpKzt2kMtObvGK85wczJBaKh3j2oeobMzB61WCZwGWvwgnBn3XhUxwwWLEfuIseRA6HUnEkdQ0ZpH7skBPtH1GnfAxnTxkroueaWxg7HuZekPweiqEcJE0jLdrqgSyYYgO1NkSQlAgIAfmUo7yp4IYAO11ZHfhsmS7HXQTZKDvFZl/shzThQ33COd6dKONHAt+0sy27Zt3LthhIcImmLfzxf3CeLVV3QXEfWM/gxy7SJHU50rwDFSyu457h6DX2c+bxfi+wmtSZVz63uqnWOoU5VrpMO7+Mi3CL58R1fbesEwseCygilSHg6xr0CfiqnkzUor3z3RQiiA5Th3kdJpUZ5Df6UFOSNlpKDJLNweoLobW1L93miG/uzWSfE9cMvnOPd7W91nemk2R2TprhFsNGZWApcafgaq67GHznPyRml6NjHcPcH3boLWhoVHMsdvVKFyQL+7InYn/fJ5cRhjXJng051d0HyF2ICDsJGhFUT1fkKu4HEwPLayN+zWNtn1EFXrUdAVFbYx2Leh7zY1nY8OWOLlcBQJoYvDllteeVI8vSw8RO7csQUa7IDWdcO/Yqtwzqpqxo7hSnnKj8EPp3Uh6iselFOgIXdVvVVG8NzTitF3qKcIiEQDsykRH/nJjDE6RpKGN9tkAYyM7XDRu+UWCXdw3qFmFDziA7Fk1tvPxdOs95E9n7kHPt5dYPk+GrpUw5OgjPduL6b7h+2CX9ZVFT Rsf4V4ZD ihtGSs54h9JO+Iv2dIEIJfnx81lP+Wkmnd+klvHqwmeD2o4+XIfVftBwm4ihMYXYgoRBTK33kRu+8aSf8pw17V8ES4DUObrqmyWUI8BVau14WtezfsrjiZFj4faeBQ05y2OtO/WqzX1ll2RdFkcisPd/IWO6IQclwuzCxCwPn+ssW+mVtrfHAUQCCHnTCP0DvFMRf98WsJSYG8iQphgow5hokr7RAqHRnqb7pMt0iW9Wu2FmfCWKW7JlfaeBs5uieH25tdX4TrLH2xoarYO1ZnzTr587VKkDNQuTT4CcZj+C8na2yYCD2yjyd388DWMDPScDFkIruOdpNbjY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: {pmd,pud,p4d}_alloc_one() is never called if the corresponding page table level is folded, as {pmd,pud,p4d}_alloc() already does the required check. We can therefore remove the runtime page table level checks in {pud,p4d}_alloc_one. The PUD helper becomes equivalent to the generic version, so we remove it altogether. This is consistent with the way arm64 and x86 handle this situation (runtime check in p4d_free() only). Signed-off-by: Kevin Brodsky --- arch/riscv/include/asm/pgalloc.h | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index f52264304f77..8ad0bbe838a2 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -12,7 +12,6 @@ #include #ifdef CONFIG_MMU -#define __HAVE_ARCH_PUD_ALLOC_ONE #define __HAVE_ARCH_PUD_FREE #include @@ -88,15 +87,6 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, } } -#define pud_alloc_one pud_alloc_one -static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - if (pgtable_l4_enabled) - return __pud_alloc_one(mm, addr); - - return NULL; -} - #define pud_free pud_free static inline void pud_free(struct mm_struct *mm, pud_t *pud) { @@ -118,15 +108,11 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, #define p4d_alloc_one p4d_alloc_one static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) { - if (pgtable_l5_enabled) { - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); - } + gfp_t gfp = GFP_PGTABLE_USER; - return NULL; + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + return (p4d_t *)get_zeroed_page(gfp); } static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) From patchwork Thu Dec 19 16:44:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 13915293 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89247E77184 for ; Thu, 19 Dec 2024 16:46:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 217C66B0099; Thu, 19 Dec 2024 11:46:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A1736B009A; Thu, 19 Dec 2024 11:46:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 01ADC6B009B; Thu, 19 Dec 2024 11:46:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D43966B0099 for ; Thu, 19 Dec 2024 11:46:38 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 81D82121791 for ; Thu, 19 Dec 2024 16:46:38 +0000 (UTC) X-FDA: 82912286898.02.2BDA896 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf03.hostedemail.com (Postfix) with ESMTP id D288020022 for ; Thu, 19 Dec 2024 16:46:21 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734626774; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KpXAr1GRKOAEyXRCUFZMJrnbzxsTSidU7pgyiULk9+Q=; b=uG3ZY5qCQWPndwq01SLCpwz1ut5ZUGVfzms2sbY8lzhcDN4qMjoWqa+Yyt3+ZbCycoGOJl g2Z4eaeK0Ccit/oBaqawRYAudlEcN9h+7j5munmrbPalsiV1g7vBEDulfB5KjQdrbB7OtO I4iVGZDo7jmqScoHWUF9AWsHiOHelWc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734626774; a=rsa-sha256; cv=none; b=jACXo05EXpYaLa5gjs2odCSM2ba1IJiMU7JjGZCNjjGrS95/eVZ9dWhToHAGNEMtW3JOu9 dlzpttz0WtEw30wCCB81QwOimHSbFz4r9Nuz5Hw4tiezar67N7CR5H2mISb2RvIpn72qYw NjLC3L2CntFue3v0Mh0kWurgf5tf/nw= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5AAB71D13; Thu, 19 Dec 2024 08:47:04 -0800 (PST) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 5BE483F58B; Thu, 19 Dec 2024 08:46:32 -0800 (PST) From: Kevin Brodsky To: linux-mm@kvack.org Cc: Kevin Brodsky , Andrew Morton , Catalin Marinas , Dave Hansen , Linus Walleij , Andy Lutomirski , Peter Zijlstra , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , linux-alpha@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-kernel@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-um@lists.infradead.org, loongarch@lists.linux.dev, x86@kernel.org Subject: [PATCH 06/10] asm-generic: pgalloc: Provide generic p4d_{alloc_one,free} Date: Thu, 19 Dec 2024 16:44:21 +0000 Message-ID: <20241219164425.2277022-7-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241219164425.2277022-1-kevin.brodsky@arm.com> References: <20241219164425.2277022-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: D288020022 X-Stat-Signature: bhk3hzrjyxh3f8wysprqkx5euogoxdpq X-Rspam-User: X-HE-Tag: 1734626781-617460 X-HE-Meta: U2FsdGVkX1/Z6Zb1tuvmxqd2ygbXjOIjJwWL9n3EXYKroM55sKrdS9vg3rtWfuWPz40B/POUuJPzMMhOmXniqhQEWdYLA+gT1ycMswGaS8y4F3hIn6cP5nnFPh8hq1cCVojmgXyLWn4nNt0qxnqBCGsVhuecgmdlRZNngqXyeePpAQL6FqVl9GwBRskakp1x92SJTqN9qEqXmzqasU7U62c5upHxwU8GZRZYbLa8QOf3frsmEGBdTLS2+iXnnChn55+Ewk/+kQheboeay2EQwLEgi8VJ0tPosh/SLKXCnuM3BfCtR7A6f9Bzn/g8wzaBw6CGwHCi2z/V8eEecAVsJLjo8+IqzpWHTCC51J/1WKwhl8qrGq4H87/rHb2PFdst2VYjVatbUWnOqUmZPgFFBG3xJYeOKYzTw1R3YWC9/Wj3dHaDM+dlukxxodofgYecwQ+1VX7I4RqLpFc1ukIb4xsG4ZSMuQgYcEcxCrVrpBrvc2b3LoUAiJXm/+c7FwaykN2T4WgPb1QqEN5cjZqC/CFSOAkiio4/AUj26jJo7NPhdnhrZdzK/8D5sh3zT1OIdOqNZAgx4w3AZdPVrNtK+Vi6BDHmhk3HX0QIh+Kf1d0HqjH+jCAw8QlHQ/wwaDDdN+eu1HBBfPHqi+7cK+QMros6j/SfK8m1WRmlD9cdn7EJQ4FjDJOdals9w/HmNe1jA+UvpNuEYd3OkFB+RP929gYBv4hQhWzIoV4eXjo7d1+F9hAzlbDBX/jqo6MPskPzuEhRl6KTIOrOqBGwQKmOIuUyYmr7Wm+7V1Tq03f0u0zDtkf+V8iSDnqsIdmiFBP6oeD/kWzxUGw14Nj5zYFNUILCYg55japhe2H9LIKrN3VyUzSkoHnyTdEcWcwb66ZkKrYzwMgsHK1KhIjN72V22+Ab8ek2S+m1gKjBZoDZ630dFD+9XEE4/3IDTREpT+0ueDOzoQdDNwTZ0GXGEwc vyY/uQCY D6eFPtkB/y+WyHMltw5IktenwfyBnetdQuY1tPhxOk7UmzNkNRZJo3zQ+caeiQs0NFhd/9AhebrC2iTTSc9oaNpTmkCMWB0Izma6/jtkGZxvIvKPPylW8oEeNhjM1A8g8NWa7nmg37UhfLw69z/aGKqvZufBtavuHKfKirXzK6Vex0k0NWEZxJyGA1YV/w1yCdo/ZhuX0twDLnvGR3CO2gZsV4bxcYVbt78vLOpWnefdXRHAk1aFzPlhHI60lkReh8T8ZhnL/fe6FUjNtdHW2u62NcbSoRNToG4ZT7fOW6puaR1RiSrtz+qVYB5ob/ufIK3su6DIl/iRqsmY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Four architectures currently implement 5-level pgtables: arm64, riscv, x86 and s390. The first three have essentially the same implementation for p4d_alloc_one() and p4d_free(), so we've got an opportunity to reduce duplication like at the lower levels. Provide a generic version of p4d_alloc_one() and p4d_free(), and make use of it on those architectures. Their implementation is the same as at PUD level, except that p4d_free() performs a runtime check by calling mm_p4d_folded(). 5-level pgtables depend on a runtime-detected hardware feature on all supported architectures, so we might as well include this check in the generic implementation. No runtime check is required in p4d_alloc_one() as the top-level p4d_alloc() already does the required check. Signed-off-by: Kevin Brodsky --- arch/arm64/include/asm/pgalloc.h | 17 ------------ arch/riscv/include/asm/pgalloc.h | 23 ---------------- arch/x86/include/asm/pgalloc.h | 18 ------------- include/asm-generic/pgalloc.h | 45 ++++++++++++++++++++++++++++++++ 4 files changed, 45 insertions(+), 58 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index e75422864d1b..2965f5a7e39e 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -85,23 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (!pgtable_l5_enabled()) - return; - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - #define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) #else static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t p4dp, pgdval_t prot) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 8ad0bbe838a2..551d614d3369 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -105,29 +105,6 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, } } -#define p4d_alloc_one p4d_alloc_one -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - -#define p4d_free p4d_free -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (pgtable_l5_enabled) - __p4d_free(mm, p4d); -} - static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index dcd836b59beb..dd4841231bb9 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -147,24 +147,6 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4 set_pgd_safe(pgd, __pgd(_PAGE_TABLE | __pa(p4d))); } -static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - gfp_t gfp = GFP_KERNEL_ACCOUNT; - - if (mm == &init_mm) - gfp &= ~__GFP_ACCOUNT; - return (p4d_t *)get_zeroed_page(gfp); -} - -static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - if (!pgtable_l5_enabled()) - return; - - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); -} - extern void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d); static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 7c48f5fbf8aa..59131629ac9c 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -215,6 +215,51 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) #endif /* CONFIG_PGTABLE_LEVELS > 3 */ +#if CONFIG_PGTABLE_LEVELS > 4 + +static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + gfp_t gfp = GFP_PGTABLE_USER; + struct ptdesc *ptdesc; + + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + gfp &= ~__GFP_HIGHMEM; + + ptdesc = pagetable_alloc_noprof(gfp, 0); + if (!ptdesc) + return NULL; + + return ptdesc_address(ptdesc); +} +#define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__)) + +#ifndef __HAVE_ARCH_P4D_ALLOC_ONE +static inline p4d_t *p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + return __p4d_alloc_one_noprof(mm, addr); +} +#define p4d_alloc_one(...) alloc_hooks(p4d_alloc_one_noprof(__VA_ARGS__)) +#endif + +static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_free(ptdesc); +} + +#ifndef __HAVE_ARCH_P4D_FREE +static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + if (!mm_p4d_folded(mm)) + __p4d_free(mm, p4d); +} +#endif + +#endif /* CONFIG_PGTABLE_LEVELS > 4 */ + #ifndef __HAVE_ARCH_PGD_FREE static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) { From patchwork Thu Dec 19 16:44:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 13915294 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E07E1E7718A for ; Thu, 19 Dec 2024 16:46:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E8C06B009B; Thu, 19 Dec 2024 11:46:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 76DB56B009C; Thu, 19 Dec 2024 11:46:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E7EC6B009D; Thu, 19 Dec 2024 11:46:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3C0886B009B for ; Thu, 19 Dec 2024 11:46:43 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 03E521C8C09 for ; Thu, 19 Dec 2024 16:46:42 +0000 (UTC) X-FDA: 82912286478.26.0156610 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf20.hostedemail.com (Postfix) with ESMTP id 9A8961C0010 for ; Thu, 19 Dec 2024 16:46:06 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf20.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734626776; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oGXtKyIBxO1nCcIBpzr+iCSsH+/WIbcJkid2JJ3TqdI=; b=SxzbzY4lyLV7mAAqDJSM9Yey8I2vzE/b7rXcjmfquylDcFCGiGmt1s2WLBC+rNzSY2yIVY m/+JegnS4lxTbIxKbAJE5wmzOBMO/EyCA8wLOPFKKiYQXir1E7uM6sT1CdlR3rt7NI1n5L 74DVM3IMoLqwsSXpNL2EEzmT21i7Fuc= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf20.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734626776; a=rsa-sha256; cv=none; b=XRq4LXCyu/sJ3vm3BN5ujD/A/NFGROJG/dsQ/xOzpDUN27tpFojn60MFufQHvIAvz/rEY0 Unn31KwCmHtTlCE3vBYYj/EDQPKx+Ob0q5NR2mVt8R07cKLpW52m/wfbOWN+4wwT+QL99V PvropFzNZQ39aOIhpiRPJLHr19Ssc8g= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8A3511480; Thu, 19 Dec 2024 08:47:08 -0800 (PST) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8C03F3F58B; Thu, 19 Dec 2024 08:46:36 -0800 (PST) From: Kevin Brodsky To: linux-mm@kvack.org Cc: Kevin Brodsky , Andrew Morton , Catalin Marinas , Dave Hansen , Linus Walleij , Andy Lutomirski , Peter Zijlstra , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , linux-alpha@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-kernel@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-um@lists.infradead.org, loongarch@lists.linux.dev, x86@kernel.org Subject: [PATCH 07/10] mm: Introduce ctor/dtor at P4D level Date: Thu, 19 Dec 2024 16:44:22 +0000 Message-ID: <20241219164425.2277022-8-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241219164425.2277022-1-kevin.brodsky@arm.com> References: <20241219164425.2277022-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9A8961C0010 X-Rspamd-Server: rspam12 X-Stat-Signature: 8afc9zx5a8ez51w71g76et8mkm58h61y X-Rspam-User: X-HE-Tag: 1734626766-710298 X-HE-Meta: U2FsdGVkX19+yT01I1lnPbKQO4r5c0g/v8oVCWpCZYVGGujG8aQcibCtsc18+2LnkLaHLt7nkaBTD4MEUS2wVugEVmYwF6ucs6xw55V+3si0KT5pIt/sUFV/exFzLY6eYtLmcwas59I6eYWsZzITH37KyaNmr7m9CpjXXW2ZFlPTC/JcYGx1aqKkEHwHK7e+7jVm0z9iI+Qw+U5Qu/LcBJ+203/8Q/AR2bNRbQeWhLfV8rXNZ1v9s8tjA0US8LRpgiriaVUF3QMQJ3CkKnTf/zD9HeXpxhsshw0xoWqivccGFrNaaCApe3TA73RQawwNM9pubSwAqAgbEULUDQwHs/S9o7IDrdAwA+T1VC+vQgZUpJWDtdsxC2vc5ucwnCGeClHe+lyzmv/uZ8xnCIx64AQHDsHL/d0YmM22PR9JOZQwhe+E1VZqHy64jx+ZyDJd0etVF6tTHVptNXFJlP05pVuMZgbqQ2unPWUmfSOKCVpodilZrc3ch7vN6p9hAiz2jhQEqJ5Sl3P9Dc19pov9c1m+/OTKSbT+tOO6ThUeL6YnIAqRnpdPlwyyEz0Bdra28vPYj01OUO+DcnEw5xd33fK4dWJq1ODl4JDF9g0CDD6I2BYbAPZy98KeTCwmZNscVNQosK2yr8tikvLOS4vJX/QNSf9Nbdif02dIl1vzSh1jNUYSJWHpiQS9AxkbIqjuDsV7Sm6NQIBmzWb+7y10OvLIRS/as+bAFL7GaqbrX7aRXUPgIeG81owwmb2MkTq8IeTFu2UdktdEpypMB86WoTwdQtGdCZ/siQdIVHCo1DTv1oYHqpq4xtsKqxUPP5HTD89AFFkbrlZ+7kCJ2Y6bMgDy59KZAzYoWMfrBXMhDlCpc2LR9bZwloLfKY/7en+fvygTaf4j3uSE6aCnNHM2OfQCUISunqb4QN2tWKNbtSV40/qOAP54dd/3ULbrSPgBs1fIoDyq4fqSxvNHUjH 3Buop4b5 t174rvoQS5O1o8vtThIKR+qz4cGQyRtzCTWMWSYFXw93X/28I2gzSYF1mMR5nFqSuYqg/RkdCFTCVXNuCJL4Coc1wZeGp9uS5w8Mx3TQQWJmIZP0pOyYt/06/JzaLWGZSE7cyZHzIqcnuuBwjVYyca8ioaK1kgL6lyj175z6uZXc197KPXr1ncG1k/sI4ykuxIHwqQ/O9tLcylDuIWa27MU+XZJsTsBkuCKAZ/8ioxEFM8yiEWfyeFcD4jg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Commit 55d2a0bd5ead ("mm: add statistics for PUD level pagetable") added accounting for PUD-level page tables. This patch does exactly the same thing for P4D-level page tables, introducing pagetable_p4d_[cd]tor with the same implementation as the PUD ctor/dtor and calling them on all alloc/free paths. Besides the small improvement in accounting accuracy, this also enables adding construction/destruction hooks in just one generic place for page tables at P4D level (like lower levels). Signed-off-by: Kevin Brodsky --- arch/riscv/include/asm/pgalloc.h | 8 ++++++-- arch/s390/include/asm/pgalloc.h | 12 ++++++++---- arch/x86/mm/pgtable.c | 3 +++ include/asm-generic/pgalloc.h | 2 ++ include/linux/mm.h | 10 ++++++++++ 5 files changed, 29 insertions(+), 6 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 551d614d3369..3c364ecc3100 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -108,8 +108,12 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) - riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + if (pgtable_l5_enabled) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); + riscv_tlb_remove_ptdesc(tlb, ptdesc); + } } #endif /* __PAGETABLE_PMD_FOLDED */ diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index 97db66ae06b9..85a5d07365aa 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -53,15 +53,19 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long address) { unsigned long *table = crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION2_ENTRY_EMPTY); + if (!table) + return NULL; + crst_table_init(table, _REGION2_ENTRY_EMPTY); + pagetable_p4d_ctor(virt_to_ptdesc(table)); return (p4d_t *) table; } static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!mm_p4d_folded(mm)) - crst_table_free(mm, (unsigned long *) p4d); + if (mm_p4d_folded(mm)) + return; + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + crst_table_free(mm, (unsigned long *) p4d); } static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 5745a354a241..c1bfdf7b4767 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -86,6 +86,9 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 59131629ac9c..bb482eeca0c3 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -230,6 +230,7 @@ static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long if (!ptdesc) return NULL; + pagetable_p4d_ctor(ptdesc); return ptdesc_address(ptdesc); } #define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__)) @@ -247,6 +248,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) struct ptdesc *ptdesc = virt_to_ptdesc(p4d); BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_p4d_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/include/linux/mm.h b/include/linux/mm.h index a5b482362792..e8b92f4bf3f1 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3161,6 +3161,16 @@ static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) __pagetable_dtor(ptdesc); } +static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) +{ + __pagetable_ctor(ptdesc); +} + +static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) +{ + __pagetable_dtor(ptdesc); +} + extern void __init pagecache_init(void); extern void free_initmem(void); From patchwork Thu Dec 19 16:44:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 13915295 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFD88E77184 for ; Thu, 19 Dec 2024 16:46:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 880186B009D; Thu, 19 Dec 2024 11:46:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 80A5C6B009E; Thu, 19 Dec 2024 11:46:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 683606B009F; Thu, 19 Dec 2024 11:46:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3B1746B009D for ; Thu, 19 Dec 2024 11:46:47 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 030311A177E for ; Thu, 19 Dec 2024 16:46:46 +0000 (UTC) X-FDA: 82912286058.01.E8B6346 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf26.hostedemail.com (Postfix) with ESMTP id 22FD3140004 for ; Thu, 19 Dec 2024 16:46:19 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=none; spf=pass (imf26.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734626782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t3fKHZ/NfJbP47FxINHB8YXuPoD7xscOU0RHBymSX5Y=; b=cSqtClpyJ1s8r7ClIy7ypdlj6sRMDZiyOAtuBzXnTHUHdm425WADPlwJBpEnLzwwpc24dH 8QYT75TU9rJzYD9Q1SnnPVTiCds35u0dprOLJME7s6pLUOtGzyVb8GE6dZa22au3Q1fRU8 HKSi2tAE0S3som227WoO4MvBCsvVRt0= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; spf=pass (imf26.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734626782; a=rsa-sha256; cv=none; b=P+1i8aJAU8Lx8PCufKjWOlbZrOv1dM/B2u5A0hvd8HOxUdQsCVNCxrvIL7fvOKrPrf1GL0 nXi8N3XZIRdI71QhgpxH2kSsJlN7mUi/WJodWFUcaYisjckWM/Hadu/YLQ4PynizzXDr8T jlXe0X/Usf/jlgW5m8fc9N/MbTM5zww= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BC7B11480; Thu, 19 Dec 2024 08:47:12 -0800 (PST) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id BB7743F58B; Thu, 19 Dec 2024 08:46:40 -0800 (PST) From: Kevin Brodsky To: linux-mm@kvack.org Cc: Kevin Brodsky , Andrew Morton , Catalin Marinas , Dave Hansen , Linus Walleij , Andy Lutomirski , Peter Zijlstra , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , linux-alpha@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-kernel@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-um@lists.infradead.org, loongarch@lists.linux.dev, x86@kernel.org Subject: [PATCH 08/10] ARM: mm: Rename PGD helpers Date: Thu, 19 Dec 2024 16:44:23 +0000 Message-ID: <20241219164425.2277022-9-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241219164425.2277022-1-kevin.brodsky@arm.com> References: <20241219164425.2277022-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Stat-Signature: fkbkzzgq3zza5ayaudn3ka7e1dm4dyjh X-Rspamd-Queue-Id: 22FD3140004 X-Rspam-User: X-HE-Tag: 1734626779-324386 X-HE-Meta: U2FsdGVkX19kuh9I+ZICsRTWAet3kTjlpnUuJ+dy0DJzlw9mUaLrcGqfu1qJNWPtq+Nhlaa5MI4G3W/oXchLo0eVH1lyVYqZaTEXCbjrZz5zl4YdcWmXffgNMkyNuLsT0Zk4hCwB/PHxKV9csro2WtTG3AmUDqMMCMzsKr3vebeZNR0+P5qc39v9ulL36jWptgriGBIhSl4U4ZEBaCnhv7qvgi+YPY+vu5B1e7DLR21gJnAOMMnA2JlzHZXWTv5xh4+1PMUZDyitR3GiVp/c11OIkHNH0tqrZX//SVnUQRURme7SbypYDBoGMUXx395c8MC59Ab04EZ4+GNcK5Tbme9EEjkU+1V1TBkEPZhyxh3F99x5Av799cnMOHaiEN32h/GuRm3mfsvvpa1L0B2+W3648CAWDIJJbl8w0u+TK4XPpBzYEfADLz9DmpKHZDzZCmU9xQxJceLqIXuM7PZOPn382N4Yr8DSu8Uxw+k0IQ2s4m+FyHpDV9v+5Tr++TO3urdC2vPmOEcZ4AMGRpeh2OAcQpVZZ4P01ziQIh0iZFgPBoD0dqZsTt3hhtWYk4HGxUNj/nKEHgqlt3XdOIOKpoExM+uW/d54wInJ0OGE3jumRuYoGCiLmU9PyzsXxkZueLxcTNl/4cePBDCa9IP8xSaLwbqt+QPBz7DjXJtKqYKYLCE4emB1DJFtrKwPmUROwHEfnAHYFLTJPAxhjBr2KPm9W+tRRjg+wMsjITwUszNZ3/xOZ5CX6DE7neLUChK0i68EIIUMh9cZQueHiOKEYTyFaR788we3uya2i2ABl+/n0ok9xstuONuXz571OQwcb9vTOh13YqZ0zg6FREn3f2yyWCykZ/l4jBp0i5q/y3EAiebQf4/0BNarDvZTOw6R/qtlx0TPiyCfTsmbvBOLuBip036SiDqI4e5gvQ3oRf6vd2T/S+or9+RktsDVhfOPxF5OYCtPa4LZbj5oqR7 WKYncmaB /lFs53CLCG71+w9iyvEYHf0l2YjaLd3+H0/tspggv2rjWrwGk6GYA2YTDwXt2bfxMVN14BAXy02fUIemJWlUGGUxKQX6IAr9rDVhpBGljGByQOL+/Nz68XnaHbhtyEJWCXRr6lejsKR00osTmgsguPZOYW872dwdlL2DPumMgTUKDJNk5kyNsisHg/O2ZwqLnDsQ+QjTeOE6BmjYJ9tL01chpqEBHIocHn9nVM+FF0TrorkMjN3T3GeYIIA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Generic implementations of __pgd_alloc and __pgd_free are about to be introduced. Rename the macros in arch/arm/mm/pgd.c to avoid clashes. While we're at it, also pass down the mm as argument to those helpers, as it will be needed to call the generic __pgd_{alloc,free}. Signed-off-by: Kevin Brodsky --- arch/arm/mm/pgd.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/arm/mm/pgd.c b/arch/arm/mm/pgd.c index f8e9bc58a84f..2a1077747758 100644 --- a/arch/arm/mm/pgd.c +++ b/arch/arm/mm/pgd.c @@ -17,11 +17,11 @@ #include "mm.h" #ifdef CONFIG_ARM_LPAE -#define __pgd_alloc() kmalloc_array(PTRS_PER_PGD, sizeof(pgd_t), GFP_KERNEL) -#define __pgd_free(pgd) kfree(pgd) +#define _pgd_alloc(mm) kmalloc_array(PTRS_PER_PGD, sizeof(pgd_t), GFP_KERNEL) +#define _pgd_free(mm, pgd) kfree(pgd) #else -#define __pgd_alloc() (pgd_t *)__get_free_pages(GFP_KERNEL, 2) -#define __pgd_free(pgd) free_pages((unsigned long)pgd, 2) +#define _pgd_alloc(mm) (pgd_t *)__get_free_pages(GFP_KERNEL, 2) +#define _pgd_free(mm, pgd) free_pages((unsigned long)pgd, 2) #endif /* @@ -35,7 +35,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm) pmd_t *new_pmd, *init_pmd; pte_t *new_pte, *init_pte; - new_pgd = __pgd_alloc(); + new_pgd = _pgd_alloc(mm); if (!new_pgd) goto no_pgd; @@ -134,7 +134,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm) no_pud: p4d_free(mm, new_p4d); no_p4d: - __pgd_free(new_pgd); + _pgd_free(mm, new_pgd); no_pgd: return NULL; } @@ -207,5 +207,5 @@ void pgd_free(struct mm_struct *mm, pgd_t *pgd_base) p4d_free(mm, p4d); } #endif - __pgd_free(pgd_base); + _pgd_free(mm, pgd_base); } From patchwork Thu Dec 19 16:44:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 13915296 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EF6FE77184 for ; Thu, 19 Dec 2024 16:46:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C78F86B009F; Thu, 19 Dec 2024 11:46:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BD9D26B00A0; Thu, 19 Dec 2024 11:46:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2D076B00A1; Thu, 19 Dec 2024 11:46:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7B43F6B009F for ; Thu, 19 Dec 2024 11:46:51 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 391C812178B for ; Thu, 19 Dec 2024 16:46:51 +0000 (UTC) X-FDA: 82912286772.30.3C6BE8C Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf02.hostedemail.com (Postfix) with ESMTP id 0AD8F80012 for ; Thu, 19 Dec 2024 16:45:45 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734626787; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZtyOjtT/A/959zxU+j8A5LReHquPkNj1E4QLGFdvTc0=; b=cIYNUbxqlx49Z5TOzoPP+HePZjpZOEG8avdZtsPfD1Hl4a/vwsBkIjh9ct5qyI8S5k3pPs JbBGx9S9ivMXgVJlZ1rBA6Ub2dpnUDPQuGJR2mvQZwkmnXmmb/vE/mfBLGYPVzCKPKOi1n zVjekUYlDtj3UW6cfs54VuSOEXrARZ8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734626787; a=rsa-sha256; cv=none; b=A9P0PQionuqL0qN3jTux/fxhhWMAV7veZsMEOnxMyWSQZHnxUQ2f5OGRtLuPGx649uU+GI treI3syNLvTf+19ykjGJ+BHr+YY+a8x7kj0bJrmAOL3ORsLBZ8HkjDhDc6u7IDyDpigeAq 8P4JuMSU/5jzy36E+DSxUuaUgSl1ZFE= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1677C1480; Thu, 19 Dec 2024 08:47:17 -0800 (PST) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id ECA773F58B; Thu, 19 Dec 2024 08:46:44 -0800 (PST) From: Kevin Brodsky To: linux-mm@kvack.org Cc: Kevin Brodsky , Andrew Morton , Catalin Marinas , Dave Hansen , Linus Walleij , Andy Lutomirski , Peter Zijlstra , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , linux-alpha@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-kernel@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-um@lists.infradead.org, loongarch@lists.linux.dev, x86@kernel.org Subject: [PATCH 09/10] asm-generic: pgalloc: Provide generic __pgd_{alloc,free} Date: Thu, 19 Dec 2024 16:44:24 +0000 Message-ID: <20241219164425.2277022-10-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241219164425.2277022-1-kevin.brodsky@arm.com> References: <20241219164425.2277022-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 0AD8F80012 X-Stat-Signature: npcmd57d6owurxkij4nu1zedoxbr66qr X-Rspam-User: X-HE-Tag: 1734626745-944982 X-HE-Meta: U2FsdGVkX1/LhMWOK47wcWR0Vv6E/Ao6vQzMWQCwSeTZW8PTyjqTAUoADTg40s9Ve6MePx8WQFyWX524pUIhXpe5YTHNtbs00WU5nv9yfQ2pYpzwQfFIIVRBxAMcqhEzEo3h0Xcn850php0uR4wnSZ0APEhwQ+lU+XCRjbrYrb+SeDM94L6zWxY8iAosXoGjqiAgHJ+hP0QrWyWMMsMuJPieVBhrX8YRIMrE5fG/I0Jg7KrfiKeW6FQWynH6W9tPrzGJ7rgV/jSQOJzTIuHqXTij8OP86bV1yNfmn4JsQ2ONUYX28c3N6q/BMFpfsA8e53WZZ626IvZfCozXvS0Vby2BfT0+T/CXmMJsb+U+dj6ulCsADvwEbA/cGnh4cFcl6qPFrD8H6W0VAQtTJweM0l5qCHgwe88bMguntaS6jx8bbpmgtVTfq5TIb7qGhAcqsO1LNwVwJAb7FfhcWWqlEuMAO2HfkH/Fo7xWgJ6mWIxFYYZHCTH7Z1QV1bUA9rD/a+2BqiPjf5OhqsgRsRjKzpfC7UR1g1b1sJ5RLxkZsYRMiG4VfmYwyOEq6dncxnD2QD9p2pLtw6W+mnM3mI7xChgDyCSYFbUGNlDSxkMci0cq5Q/Dh/WN6LBEIYWCMNTrcBwH+vInozFXSBmoHwlRY/zDn4gpwkYA44pyygSKJjDQ+e2DzHZz1VOh94uPHhnaquIMQdaQkwWiHJ3zjO1Nu0oiLBi/uKJz3gx37RcUDhYqB/PaBTY7hIdEROdcmMpNMBFWfYtuXQO7zCaYx2/b4IL6f2JFp1Kuv+rnXxHWxbte5spkilvKknnHm0Sk/k+8SgWrZ9rp9rP1UoE9iPkrJuVwzXELscsFaUaqce+fMbf+GS6KBVF3fdqJB5/r+a17nzWdZm/6O6594vfqtopWAZZcbH5L0jXRBwHox+odzZnacSgyqPYayVfLyd10VbyEJjxtwcynQPLKia0BH/q 2axG0jR2 BeU+U1RltQVi5mOCliWtJiSXe2tyl/ag2sIbldnlLpAT85PIrBcHDPtyzLdKVvzfCgszews8AS/D1RwsmROvqyfAqTdtgYk4aqt5uedO44MLZU0z8A0iT6b7WLoOxgQ+f1x4/NHIFR9+ZQO4cTPwjs/3ZTfbyJfpsIG+TsgQWCpNYQdD8Re02UlvBQ97axeiPBUUJNsKtNuqh8TVxU8dT+k5HjEzCjdkxgDi4eEG9RMx4w8KXpyMlxBd4UzJzM+W0QB0pnAulZcpwW+Vtc3IMO1ZBJVmGcrjapJ7AGui78LBX7fBfXYonbZUB9xVv237IHpFTpC9C6r6y5mI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We already have a generic implementation of alloc/free up to P4D level, as well as pgd_free(). Let's finish the work and add a generic PGD-level alloc helper as well. Unlike at lower levels, almost all architectures need some specific magic at PGD level (typically initialising PGD entries), so introducing a generic pgd_alloc() isn't worth it. Instead we introduce two new helpers, __pgd_alloc() and __pgd_free(), and make use of them in the arch-specific pgd_alloc() and pgd_free() wherever possible. To accommodate as many arch as possible, __pgd_alloc() takes a page alocation order. Because pagetable_alloc() allocates zeroed pages, we are also able to get rid of zeroing code in some implementations of pgd_alloc(). Some trivial implementations of pgd_free() also become unnecessary once __pgd_alloc() is used; remove them. Another small improvement is consistent accounting of PGD pages by using GFP_PGTABLE_{USER,KERNEL} as appropriate. Not all PGD allocations can be handled by the generic helpers. In particular, multiple architectures allocate PGDs from a kmem_cache, and those PGDs may not be page-sized. Signed-off-by: Kevin Brodsky --- arch/alpha/mm/init.c | 2 +- arch/arc/include/asm/pgalloc.h | 9 ++------- arch/arm/mm/pgd.c | 8 +++----- arch/arm64/mm/pgd.c | 4 ++-- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/mm/pgtable.c | 7 +++---- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/microblaze/include/asm/pgalloc.h | 7 +------ arch/mips/include/asm/pgalloc.h | 6 ------ arch/mips/mm/pgtable.c | 8 +++----- arch/nios2/mm/pgtable.c | 3 ++- arch/openrisc/include/asm/pgalloc.h | 6 ++---- arch/parisc/include/asm/pgalloc.h | 16 +--------------- arch/riscv/include/asm/pgalloc.h | 3 +-- arch/um/kernel/mem.c | 7 +++---- arch/x86/mm/pgtable.c | 24 +++++++++++------------- arch/xtensa/include/asm/pgalloc.h | 2 +- include/asm-generic/pgalloc.h | 27 ++++++++++++++++++++++++++- 19 files changed, 65 insertions(+), 80 deletions(-) diff --git a/arch/alpha/mm/init.c b/arch/alpha/mm/init.c index 4fe618446e4c..61c2198b1359 100644 --- a/arch/alpha/mm/init.c +++ b/arch/alpha/mm/init.c @@ -42,7 +42,7 @@ pgd_alloc(struct mm_struct *mm) { pgd_t *ret, *init; - ret = (pgd_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO); + ret = __pgd_alloc(mm, 0); init = pgd_offset(&init_mm, 0UL); if (ret) { #ifdef CONFIG_ALPHA_LARGE_VMALLOC diff --git a/arch/arc/include/asm/pgalloc.h b/arch/arc/include/asm/pgalloc.h index 096b8ef58edb..dfae070fe8d5 100644 --- a/arch/arc/include/asm/pgalloc.h +++ b/arch/arc/include/asm/pgalloc.h @@ -53,19 +53,14 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, pgtable_t pte_ static inline pgd_t *pgd_alloc(struct mm_struct *mm) { - pgd_t *ret = (pgd_t *) __get_free_page(GFP_KERNEL); + pgd_t *ret = __pgd_alloc(mm, 0); if (ret) { int num, num2; - num = USER_PTRS_PER_PGD + USER_KERNEL_GUTTER / PGDIR_SIZE; - memzero(ret, num * sizeof(pgd_t)); + num = USER_PTRS_PER_PGD + USER_KERNEL_GUTTER / PGDIR_SIZE; num2 = VMALLOC_SIZE / PGDIR_SIZE; memcpy(ret + num, swapper_pg_dir + num, num2 * sizeof(pgd_t)); - - memzero(ret + num + num2, - (PTRS_PER_PGD - num - num2) * sizeof(pgd_t)); - } return ret; } diff --git a/arch/arm/mm/pgd.c b/arch/arm/mm/pgd.c index 2a1077747758..4eb81b7ed03a 100644 --- a/arch/arm/mm/pgd.c +++ b/arch/arm/mm/pgd.c @@ -17,11 +17,11 @@ #include "mm.h" #ifdef CONFIG_ARM_LPAE -#define _pgd_alloc(mm) kmalloc_array(PTRS_PER_PGD, sizeof(pgd_t), GFP_KERNEL) +#define _pgd_alloc(mm) kmalloc_array(PTRS_PER_PGD, sizeof(pgd_t), GFP_KERNEL | __GFP_ZERO) #define _pgd_free(mm, pgd) kfree(pgd) #else -#define _pgd_alloc(mm) (pgd_t *)__get_free_pages(GFP_KERNEL, 2) -#define _pgd_free(mm, pgd) free_pages((unsigned long)pgd, 2) +#define _pgd_alloc(mm) __pgd_alloc(mm, 2) +#define _pgd_free(mm, pgd) __pgd_free(mm, pgd) #endif /* @@ -39,8 +39,6 @@ pgd_t *pgd_alloc(struct mm_struct *mm) if (!new_pgd) goto no_pgd; - memset(new_pgd, 0, USER_PTRS_PER_PGD * sizeof(pgd_t)); - /* * Copy over the kernel and IO PGD entries */ diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c index 0c501cabc238..8160cff35089 100644 --- a/arch/arm64/mm/pgd.c +++ b/arch/arm64/mm/pgd.c @@ -33,7 +33,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm) gfp_t gfp = GFP_PGTABLE_USER; if (pgdir_is_page_size()) - return (pgd_t *)__get_free_page(gfp); + return __pgd_alloc(mm, 0); else return kmem_cache_alloc(pgd_cache, gfp); } @@ -41,7 +41,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm) void pgd_free(struct mm_struct *mm, pgd_t *pgd) { if (pgdir_is_page_size()) - free_page((unsigned long)pgd); + __pgd_free(mm, pgd); else kmem_cache_free(pgd_cache, pgd); } diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgalloc.h index 9c84c9012e53..9a3bbf16f03f 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -44,7 +44,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) pgd_t *ret; pgd_t *init; - ret = (pgd_t *) __get_free_page(GFP_KERNEL); + ret = __pgd_alloc(mm, 0); if (ret) { init = pgd_offset(&init_mm, 0UL); pgd_init((unsigned long *)ret); diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/pgalloc.h index 55988625e6fb..289dace6d76d 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -22,7 +22,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) { pgd_t *pgd; - pgd = (pgd_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO); + pgd = __pgd_alloc(mm, 0); /* * There may be better ways to do this, but to ensure diff --git a/arch/loongarch/mm/pgtable.c b/arch/loongarch/mm/pgtable.c index 3fa69b23ff84..22a94bb3e6e8 100644 --- a/arch/loongarch/mm/pgtable.c +++ b/arch/loongarch/mm/pgtable.c @@ -23,11 +23,10 @@ EXPORT_SYMBOL(tlb_virt_to_page); pgd_t *pgd_alloc(struct mm_struct *mm) { - pgd_t *init, *ret = NULL; - struct ptdesc *ptdesc = pagetable_alloc(GFP_KERNEL & ~__GFP_HIGHMEM, 0); + pgd_t *init, *ret; - if (ptdesc) { - ret = (pgd_t *)ptdesc_address(ptdesc); + ret = __pgd_alloc(mm, 0); + if (ret) { init = pgd_offset(&init_mm, 0UL); pgd_init(ret); memcpy(ret + USER_PTRS_PER_PGD, init + USER_PTRS_PER_PGD, diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/sun3_pgalloc.h index 4a137eecb6fe..e91b0133df5d 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -43,7 +43,7 @@ static inline pgd_t * pgd_alloc(struct mm_struct *mm) { pgd_t *new_pgd; - new_pgd = (pgd_t *)get_zeroed_page(GFP_KERNEL); + new_pgd = __pgd_alloc(mm, 0); memcpy(new_pgd, swapper_pg_dir, PAGE_SIZE); memset(new_pgd, 0, (PAGE_OFFSET >> PGDIR_SHIFT)); return new_pgd; diff --git a/arch/microblaze/include/asm/pgalloc.h b/arch/microblaze/include/asm/pgalloc.h index 6c33b05f730f..084a8a0dc239 100644 --- a/arch/microblaze/include/asm/pgalloc.h +++ b/arch/microblaze/include/asm/pgalloc.h @@ -21,12 +21,7 @@ extern void __bad_pte(pmd_t *pmd); -static inline pgd_t *get_pgd(void) -{ - return (pgd_t *)__get_free_pages(GFP_KERNEL|__GFP_ZERO, 0); -} - -#define pgd_alloc(mm) get_pgd() +#define pgd_alloc(mm) __pgd_alloc(mm, 0) extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h index f4440edcd8fe..9ee8426b2e9d 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -15,7 +15,6 @@ #define __HAVE_ARCH_PMD_ALLOC_ONE #define __HAVE_ARCH_PUD_ALLOC_ONE -#define __HAVE_ARCH_PGD_FREE #include static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, @@ -49,11 +48,6 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) extern void pgd_init(void *addr); extern pgd_t *pgd_alloc(struct mm_struct *mm); -static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) -{ - pagetable_free(virt_to_ptdesc(pgd)); -} - #define __pte_free_tlb(tlb, pte, address) \ do { \ pagetable_pte_dtor(page_ptdesc(pte)); \ diff --git a/arch/mips/mm/pgtable.c b/arch/mips/mm/pgtable.c index 1506e458040d..10835414819f 100644 --- a/arch/mips/mm/pgtable.c +++ b/arch/mips/mm/pgtable.c @@ -10,12 +10,10 @@ pgd_t *pgd_alloc(struct mm_struct *mm) { - pgd_t *init, *ret = NULL; - struct ptdesc *ptdesc = pagetable_alloc(GFP_KERNEL & ~__GFP_HIGHMEM, - PGD_TABLE_ORDER); + pgd_t *init, *ret; - if (ptdesc) { - ret = ptdesc_address(ptdesc); + ret = __pgd_alloc(mm, PGD_TABLE_ORDER); + if (ret) { init = pgd_offset(&init_mm, 0UL); pgd_init(ret); memcpy(ret + USER_PTRS_PER_PGD, init + USER_PTRS_PER_PGD, diff --git a/arch/nios2/mm/pgtable.c b/arch/nios2/mm/pgtable.c index 7c76e8a7447a..6470ed378782 100644 --- a/arch/nios2/mm/pgtable.c +++ b/arch/nios2/mm/pgtable.c @@ -11,6 +11,7 @@ #include #include +#include /* pteaddr: * ptbase | vpn* | zero @@ -54,7 +55,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm) { pgd_t *ret, *init; - ret = (pgd_t *) __get_free_page(GFP_KERNEL); + ret = __pgd_alloc(mm, 0); if (ret) { init = pgd_offset(&init_mm, 0UL); pgd_init(ret); diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/asm/pgalloc.h index c6a73772a546..c068c4942467 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -41,15 +41,13 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, */ static inline pgd_t *pgd_alloc(struct mm_struct *mm) { - pgd_t *ret = (pgd_t *)__get_free_page(GFP_KERNEL); + pgd_t *ret = __pgd_alloc(mm, 0); - if (ret) { - memset(ret, 0, USER_PTRS_PER_PGD * sizeof(pgd_t)); + if (ret) memcpy(ret + USER_PTRS_PER_PGD, swapper_pg_dir + USER_PTRS_PER_PGD, (PTRS_PER_PGD - USER_PTRS_PER_PGD) * sizeof(pgd_t)); - } return ret; } diff --git a/arch/parisc/include/asm/pgalloc.h b/arch/parisc/include/asm/pgalloc.h index 3e8dbd79670b..2ca74a56415c 100644 --- a/arch/parisc/include/asm/pgalloc.h +++ b/arch/parisc/include/asm/pgalloc.h @@ -11,26 +11,12 @@ #include #define __HAVE_ARCH_PMD_ALLOC_ONE -#define __HAVE_ARCH_PGD_FREE #include /* Allocate the top level pgd (page directory) */ static inline pgd_t *pgd_alloc(struct mm_struct *mm) { - pgd_t *pgd; - - pgd = (pgd_t *) __get_free_pages(GFP_KERNEL, PGD_TABLE_ORDER); - if (unlikely(pgd == NULL)) - return NULL; - - memset(pgd, 0, PAGE_SIZE << PGD_TABLE_ORDER); - - return pgd; -} - -static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) -{ - free_pages((unsigned long)pgd, PGD_TABLE_ORDER); + return __pgd_alloc(mm, PGD_TABLE_ORDER); } #if CONFIG_PGTABLE_LEVELS == 3 diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 3c364ecc3100..d527f141be1c 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -128,9 +128,8 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) { pgd_t *pgd; - pgd = (pgd_t *)__get_free_page(GFP_KERNEL); + pgd = __pgd_alloc(mm, 0); if (likely(pgd != NULL)) { - memset(pgd, 0, USER_PTRS_PER_PGD * sizeof(pgd_t)); /* Copy kernel mappings */ sync_kernel_mappings(pgd); } diff --git a/arch/um/kernel/mem.c b/arch/um/kernel/mem.c index 53248ed04771..d98812907493 100644 --- a/arch/um/kernel/mem.c +++ b/arch/um/kernel/mem.c @@ -214,14 +214,13 @@ void free_initmem(void) pgd_t *pgd_alloc(struct mm_struct *mm) { - pgd_t *pgd = (pgd_t *)__get_free_page(GFP_KERNEL); + pgd_t *pgd = __pgd_alloc(mm, 0); - if (pgd) { - memset(pgd, 0, USER_PTRS_PER_PGD * sizeof(pgd_t)); + if (pgd) memcpy(pgd + USER_PTRS_PER_PGD, swapper_pg_dir + USER_PTRS_PER_PGD, (PTRS_PER_PGD - USER_PTRS_PER_PGD) * sizeof(pgd_t)); - } + return pgd; } diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index c1bfdf7b4767..00917ef609b6 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -395,15 +395,14 @@ void __init pgtable_cache_init(void) SLAB_PANIC, NULL); } -static inline pgd_t *_pgd_alloc(void) +static inline pgd_t *_pgd_alloc(struct mm_struct *mm) { /* * If no SHARED_KERNEL_PMD, PAE kernel is running as a Xen domain. * We allocate one page for pgd. */ if (!SHARED_KERNEL_PMD) - return (pgd_t *)__get_free_pages(GFP_PGTABLE_USER, - PGD_ALLOCATION_ORDER); + return __pgd_alloc(mm, PGD_ALLOCATION_ORDER); /* * Now PAE kernel is not running as a Xen domain. We can allocate @@ -412,24 +411,23 @@ static inline pgd_t *_pgd_alloc(void) return kmem_cache_alloc(pgd_cache, GFP_PGTABLE_USER); } -static inline void _pgd_free(pgd_t *pgd) +static inline void _pgd_free(struct mm_struct *mm, pgd_t *pgd) { if (!SHARED_KERNEL_PMD) - free_pages((unsigned long)pgd, PGD_ALLOCATION_ORDER); + __pgd_free(mm, pgd); else kmem_cache_free(pgd_cache, pgd); } #else -static inline pgd_t *_pgd_alloc(void) +static inline pgd_t *_pgd_alloc(struct mm_struct *mm) { - return (pgd_t *)__get_free_pages(GFP_PGTABLE_USER, - PGD_ALLOCATION_ORDER); + return __pgd_alloc(mm, PGD_ALLOCATION_ORDER); } -static inline void _pgd_free(pgd_t *pgd) +static inline void _pgd_free(struct mm_struct *mm, pgd_t *pgd) { - free_pages((unsigned long)pgd, PGD_ALLOCATION_ORDER); + __pgd_free(mm, pgd); } #endif /* CONFIG_X86_PAE */ @@ -439,7 +437,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm) pmd_t *u_pmds[MAX_PREALLOCATED_USER_PMDS]; pmd_t *pmds[MAX_PREALLOCATED_PMDS]; - pgd = _pgd_alloc(); + pgd = _pgd_alloc(mm); if (pgd == NULL) goto out; @@ -482,7 +480,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm) if (sizeof(pmds) != 0) free_pmds(mm, pmds, PREALLOCATED_PMDS); out_free_pgd: - _pgd_free(pgd); + _pgd_free(mm, pgd); out: return NULL; } @@ -492,7 +490,7 @@ void pgd_free(struct mm_struct *mm, pgd_t *pgd) pgd_mop_up_pmds(mm, pgd); pgd_dtor(pgd); paravirt_pgd_free(mm, pgd); - _pgd_free(pgd); + _pgd_free(mm, pgd); } /* diff --git a/arch/xtensa/include/asm/pgalloc.h b/arch/xtensa/include/asm/pgalloc.h index 7fc0f9126dd3..1919ee9c3dd6 100644 --- a/arch/xtensa/include/asm/pgalloc.h +++ b/arch/xtensa/include/asm/pgalloc.h @@ -29,7 +29,7 @@ static inline pgd_t* pgd_alloc(struct mm_struct *mm) { - return (pgd_t*) __get_free_page(GFP_KERNEL | __GFP_ZERO); + return __pgd_alloc(mm, 0); } static inline void ptes_clear(pte_t *ptep) diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index bb482eeca0c3..daa8bea36952 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -262,10 +262,35 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) #endif /* CONFIG_PGTABLE_LEVELS > 4 */ +static inline pgd_t *__pgd_alloc_noprof(struct mm_struct *mm, unsigned int order) +{ + gfp_t gfp = GFP_PGTABLE_USER; + struct ptdesc *ptdesc; + + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + gfp &= ~__GFP_HIGHMEM; + + ptdesc = pagetable_alloc_noprof(gfp, order); + if (!ptdesc) + return NULL; + + return ptdesc_address(ptdesc); +} +#define __pgd_alloc(...) alloc_hooks(__pgd_alloc_noprof(__VA_ARGS__)) + +static inline void __pgd_free(struct mm_struct *mm, pgd_t *pgd) +{ + struct ptdesc *ptdesc = virt_to_ptdesc(pgd); + + BUG_ON((unsigned long)pgd & (PAGE_SIZE-1)); + pagetable_free(ptdesc); +} + #ifndef __HAVE_ARCH_PGD_FREE static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) { - pagetable_free(virt_to_ptdesc(pgd)); + __pgd_free(mm, pgd); } #endif From patchwork Thu Dec 19 16:44:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 13915297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C515E77184 for ; Thu, 19 Dec 2024 16:46:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1097F6B00A1; Thu, 19 Dec 2024 11:46:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 067FF6B00A2; Thu, 19 Dec 2024 11:46:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DFD7F6B00A3; Thu, 19 Dec 2024 11:46:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BA0ED6B00A1 for ; Thu, 19 Dec 2024 11:46:55 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 781D8817C1 for ; Thu, 19 Dec 2024 16:46:55 +0000 (UTC) X-FDA: 82912287234.05.BC075A6 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf14.hostedemail.com (Postfix) with ESMTP id 0F1F510001C for ; Thu, 19 Dec 2024 16:46:16 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734626791; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SXme/GytDIOL4n/9MQ9Cf0OJewNEVpT4E5Wg5Chg+u0=; b=MtfUpx8VgjDQiX557wyxHgM57aWvBZBGbbWotryHsRBCTFWOYJhVN5xQ6vsOsMbLNXzZe1 E7T9g4PNCcRcXX5tF4G0cDxURNsEDJzAm0Uc8U2a4v7GeM3vmulS0Si//nKSVVE+8oNgh1 hbPPaZppGtSNVBi5Vs+9SHP9vwPh0no= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734626791; a=rsa-sha256; cv=none; b=M+B1QQmE3cs5Tb9+nrg7eiIdwbnPIhXmwqjXqgTZczyicPimeHnu6ZncONjedG7g11tD4P 4nl+AjEJBua1BverObPacZW1atuDv0wix+ZNVtlIEAjFgxKSFfPlJlw4hi6ADHlscJkT1+ bOJBFvhgT0xTGMjv7NrN/ZLW9J0AvWk= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 46F061480; Thu, 19 Dec 2024 08:47:21 -0800 (PST) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 47EE93F58B; Thu, 19 Dec 2024 08:46:49 -0800 (PST) From: Kevin Brodsky To: linux-mm@kvack.org Cc: Kevin Brodsky , Andrew Morton , Catalin Marinas , Dave Hansen , Linus Walleij , Andy Lutomirski , Peter Zijlstra , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , linux-alpha@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-kernel@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-um@lists.infradead.org, loongarch@lists.linux.dev, x86@kernel.org Subject: [PATCH 10/10] mm: Introduce ctor/dtor at PGD level Date: Thu, 19 Dec 2024 16:44:25 +0000 Message-ID: <20241219164425.2277022-11-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241219164425.2277022-1-kevin.brodsky@arm.com> References: <20241219164425.2277022-1-kevin.brodsky@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Stat-Signature: p1x71j3jxgffcjgtnfptic861x8iff77 X-Rspamd-Queue-Id: 0F1F510001C X-Rspam-User: X-HE-Tag: 1734626776-382222 X-HE-Meta: U2FsdGVkX19hZboZ6FKGdQlQpWfSlyLwLpHSMrPRti+l36iIQs2ThzmaZNqA7L70tvgLabn+7Ly8pAojshityVnOnzx+jdhvIoKX2cf31HnNyNa3TFwmuq2ccBa8XMrQIB3KQpMgU1XFz17RR3jjhUbwHn+g+1s6y8NbVlLTkypnsRdL5gVNVvB8WGNhrySNpTEgfNf0kL2mrd8PmOTZUmpWd9VpKgwEni9HpNvcMnHYrr+qgUuv+ETlmu1UAvmP042fwFaAvF0JtnGBMZqx3sXDV5V+nXEepf8/jzt/wd5m53IW8ug3dX7TSlgNnQ1pwveHZWMi5mkDOFMrkxxprKzXxT8zCCOL76cRG4KSzZoIR8P/B+uYp09rf8WX1q//RqzBK2DQM1z+TjbPim5ykK0kKEmfSWdK4bHq0z+YpDvijsIgnKJD+5r2KWdkbpDffC/J29Blq95oJS9iDnlM6MMXORvvFQgP/HhOV5Lr3hGOysPHXrzEUvzikIJbdwwHttcm2CsnaeICsL/0wZYi1R88N3D4jC745f0fS/V0pMwDFrWsUH/DZipkHPpIAc2wiTxEadjczGi2DwjHEM2PzHpUxAYpxK87Y8VfJA0b51jpsBpc9/dTjBAEAtf92CZxgAGIgCaC7+aPMiITY94tTOUPmrpKCDQ7Yz2KehlXdQu0u/xY/2dY5cPDySeMMkrqcG6LfGW0i+z6FcYyE0Yz+FSLVZ7i6OMw7m06mA9+NCxVEYQGezEUN9fiIRu5YiR1EoMSseZXkECQGlMVCjxFlFYhAxg+FNGEz6srspJT+8KGBJv/rBcnNgjsKal7cOJRnxXmoF41nC5zvox2D65qLOrMxR/q8yhjgvJGDRGyFI3vkMUcaJ/7pPEVIBkt6Gd7EbCPFFQMlqGhtdCmOyMUignStf+G9VxNVuu8g5GwcafqcIwirJdh0qYf57SnaIuBjZU5cT/i7C6KRImxGKo 4e0TzBny +qcd0TyoEhPevEArI7TK25AxqGQ1jSBhgIRzp+p+CxXk9gB2OSAV2UqTbAP/v5bFdKXfEgPw9bztZLrl9sy+yshP3eVOk1Rf8Q4zsvrxP8azgwCRxV8bzM1uZCsu9tmTOWU1MyqD6ujFrRfjPcVfg91GYISuJU2QyQXgI5MYZ19o77O0/cguuX9S0b+WZ9u/e5jH7VPMzXaRsViM/QtaJreKHSHVEFIH2fGn+BuiZcuGEYTPI7DT/GEmfHQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Following on from the introduction of P4D-level ctor/dtor, let's finish the job and introduce ctor/dtor at PGD level. The incurred improvement in page accounting is minimal - the main motivation is to create a single, generic place where construction/destruction hooks can be added for all page table pages. This patch should cover all architectures and all configurations where PGDs are one or more regular pages. This excludes any configuration where PGDs are allocated from a kmem_cache object. Signed-off-by: Kevin Brodsky --- arch/m68k/include/asm/mcf_pgalloc.h | 2 ++ arch/m68k/mm/motorola.c | 6 ++++++ arch/s390/include/asm/pgalloc.h | 8 +++++++- include/asm-generic/pgalloc.h | 2 ++ include/linux/mm.h | 10 ++++++++++ 5 files changed, 27 insertions(+), 1 deletion(-) diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mcf_pgalloc.h index 302c5bf67179..7bb9652e1d67 100644 --- a/arch/m68k/include/asm/mcf_pgalloc.h +++ b/arch/m68k/include/asm/mcf_pgalloc.h @@ -73,6 +73,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t pgtable) static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) { + pagetable_pgd_dtor(virt_to_ptdesc(pgd)); pagetable_free(virt_to_ptdesc(pgd)); } @@ -84,6 +85,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) if (!ptdesc) return NULL; + pagetable_pgd_ctor(ptdesc); new_pgd = ptdesc_address(ptdesc); memcpy(new_pgd, swapper_pg_dir, PTRS_PER_PGD * sizeof(pgd_t)); diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index 37010ee15928..b0fbb369589f 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -169,6 +169,9 @@ void *get_pointer_table(int type) case TABLE_PMD: pagetable_pmd_ctor(virt_to_ptdesc(page)); break; + case TABLE_PGD: + pagetable_pgd_ctor(virt_to_ptdesc(page)); + break; } mmu_page_ctor(page); @@ -215,6 +218,9 @@ int free_pointer_table(void *table, int type) case TABLE_PMD: pagetable_pmd_dtor(virt_to_ptdesc((void *)page)); break; + case TABLE_PGD: + pagetable_pgd_dtor(virt_to_ptdesc((void *)page)); + break; } free_page (page); diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index 85a5d07365aa..00b69f4ddf17 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -126,11 +126,17 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) static inline pgd_t *pgd_alloc(struct mm_struct *mm) { - return (pgd_t *) crst_table_alloc(mm); + unsigned long *table = crst_table_alloc(mm); + + if (!table) + return NULL; + pagetable_pgd_ctor(virt_to_ptdesc(table)); + return (pgd_t *) table; } static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) { + pagetable_pgd_dtor(virt_to_ptdesc(pgd)); crst_table_free(mm, (unsigned long *) pgd); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index daa8bea36952..112b09dc992e 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -275,6 +275,7 @@ static inline pgd_t *__pgd_alloc_noprof(struct mm_struct *mm, unsigned int order if (!ptdesc) return NULL; + pagetable_pgd_ctor(ptdesc); return ptdesc_address(ptdesc); } #define __pgd_alloc(...) alloc_hooks(__pgd_alloc_noprof(__VA_ARGS__)) @@ -284,6 +285,7 @@ static inline void __pgd_free(struct mm_struct *mm, pgd_t *pgd) struct ptdesc *ptdesc = virt_to_ptdesc(pgd); BUG_ON((unsigned long)pgd & (PAGE_SIZE-1)); + pagetable_pgd_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/include/linux/mm.h b/include/linux/mm.h index e8b92f4bf3f1..7347da3460c5 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3171,6 +3171,16 @@ static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) __pagetable_dtor(ptdesc); } +static inline void pagetable_pgd_ctor(struct ptdesc *ptdesc) +{ + __pagetable_ctor(ptdesc); +} + +static inline void pagetable_pgd_dtor(struct ptdesc *ptdesc) +{ + __pagetable_dtor(ptdesc); +} + extern void __init pagecache_init(void); extern void free_initmem(void);