From patchwork Mon Sep 18 06:31:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13388844 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C938AC46CA1 for ; Mon, 18 Sep 2023 06:32:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 58BC46B0251; Mon, 18 Sep 2023 02:32:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 53B976B0252; Mon, 18 Sep 2023 02:32:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42A606B0253; Mon, 18 Sep 2023 02:32:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 34DC86B0251 for ; Mon, 18 Sep 2023 02:32:02 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 153A9160B96 for ; Mon, 18 Sep 2023 06:32:02 +0000 (UTC) X-FDA: 81248748084.09.5C033A2 Received: from out30-124.freemail.mail.aliyun.com (out30-124.freemail.mail.aliyun.com [115.124.30.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 317D818001F for ; Mon, 18 Sep 2023 06:31:58 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf06.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.124 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695018720; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=XQuyMN8O3oiBq19FjoK/pMViXtzsrJa0BVIEwqRR4+Q=; b=SlAIKZ1hf1J7JzshhHdmhKME8L5/FEmEBxeYjsg5lpoXWUFx3Zs/pnYqgaTAxOio6ilSfe edBBCempowXXcXzt0YYEh+BRhPu79xfhOBB0g6k5e6mFFvYnth8Xdiw73FzIftc+aSr3yj tfOV58qXqMWgzLmXyaOIwJxLFK1WD+k= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf06.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.124 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695018720; a=rsa-sha256; cv=none; b=BGAl9MhinXVE3vbvSanhZbyDw4Sn28FkBdcdrQc1y5iMlwRvDXuLxzheEDZ3RsPX5qvNFs 8fwUiSQ5NgWQabXHwaz2YPeJRK/YgTB12hFuFLaKRPdSa/9RsQDnNOjlBTQTNoQJtI0Oit tcrYEiYzRof66UqJTCQIMXfOWvzp7lg= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R541e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046060;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=22;SR=0;TI=SMTPD_---0VsH48PF_1695018713; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VsH48PF_1695018713) by smtp.aliyun-inc.com; Mon, 18 Sep 2023 14:31:54 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: will@kernel.org, aneesh.kumar@linux.ibm.com, npiggin@gmail.com, peterz@infradead.org, catalin.marinas@arm.com, chenhuacai@kernel.org, tsbogend@alpha.franken.de, dave.hansen@linux.intel.com, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, arnd@arndb.de, willy@infradead.org, baolin.wang@linux.alibaba.com, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org Subject: [PATCH] mm: add statistics for PUD level pagetable Date: Mon, 18 Sep 2023 14:31:42 +0800 Message-Id: <876c71c03a7e69c17722a690e3225a4f7b172fb2.1695017383.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: hi9tbbpiuzj8wtmacwmn589s7pnmja9w X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 317D818001F X-HE-Tag: 1695018718-904698 X-HE-Meta: U2FsdGVkX19tX4xHUae0g7C6i2yzcRGONgPuldKMmY1yK+ChtB+vI7xsnZgih3jZmCRpvBiweJSx07hZ7UpQBiZnjavgnjtfa+UUNxqewMK8tjyQtuDcGu8lt01LvYGIjF/lSvZdOWvgRRPLF/4YiYlJR/CvGdyV+n/h4pmVZ+dTpVS5u5v3wtx/BKEECrvKoOMWjbqYoWWV5ppaik1ngSRE6LkdCjuWh5shZ+vDTRiPZpxqthn85c7aCgMsqQLWrtpR7B5LeZzQFJib5PNVCfmebNbCs5/ozW01JQzIWIxmF8inaOs6gyOTybk9jaPlHeIjgQ7PP97vc6cObnEcpAxXqZFeXE1xubNwL2h7nkIlasZLoDbhtp5vf8xWnMuy2S7w1wTexx3fbPFT/7hyBMkPKlxE27uGlVO3ZD8fjcoGTobQI7H2mPVgxPPfojxeqJEuPYBIuXrMzyU5kxvwiOvqLIjgMfnnFJKkA/usWMkI8uRcoK3x/i+3FLMjmhCGpfa2F05DoV8gn7v/TkEyxkTt+S9BBdkpqinWWxZalEDetDBrGxf6wCnYtJTp5Yk8wG5lmnDRE90Vq5Alaok770Yy/LYHs/tMtlSBnxvu5Zc1JM9GYl0mzmDV0+Dd0AW5JHoKTrfFiVgZB1msiU2kgMJ76OU3UIkt99NVJcxBHWD8YpUJpXlu+6kFgh77T1rQMvHtEQ/yzTW5XuEzOWtr3Yo9iB4ThLQKWif4SgZisWsH7C6hR+jVA6MIjweVPy6XJHcwlUxhzoL/l+Su5cpGqxWeRUOi+N7ke2K1WiPKpH0cgaMF6KlJeF4sAuIz4NCSzynyiGj/nPvYNtbD7nm+v273iPRb/xkemyvbHDL3KIVWjX7hrN2j5CTuDQktbt8zhCuiv2U4VY55Ud+yKUzdfJ/hLqGYpjIRrNja3IbVGb/ZrzeHJgXdqI8XPfmG68FqqJ20dhTut1TgwTxOX69 srCnKIPA oK+YI6IovNdpo0vXLxIc7hu7taxDCfodbHz0wzdXS2qiYygt+e15cBLbwt5b/IcQeRhJcKRHBpb6tqAQ6buyo2T9d9oXpW8M34e6O X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Recently, we found that cross-die access to pagetable pages on ARM64 machines can cause performance fluctuations in our business. Currently, there are no PMU events available to track this situation on our ARM64 machines, so an accurate pagetable accounting can help to analyze this issue, but now the PUD level pagetable accounting is missed. So introducing pagetable_pud_ctor/dtor() to help to get an accurate PUD pagetable accounting, as well as converting the architectures with using generic PUD pagatable allocation to add corresponding PUD pagetable accounting. Moreover this patch will also mark the PUD level pagetable with PG_table flag, which will help to do sanity validation in unpoison_memory(). On my testing machine, I can see more pagetables statistics after the patch with page-types tool: Before patch: flags page-count MB symbolic-flags long-symbolic-flags 0x0000000004000000 27326 106 __________________________g_________________ pgtable After patch: 0x0000000004000000 27541 107 __________________________g_________________ pgtable Signed-off-by: Baolin Wang Acked-by: Mike Rapoport (IBM) Acked-by: Vishal Moola (Oracle) --- arch/arm64/include/asm/tlb.h | 5 ++++- arch/loongarch/include/asm/pgalloc.h | 1 + arch/mips/include/asm/pgalloc.h | 1 + arch/x86/mm/pgtable.c | 3 +++ include/asm-generic/pgalloc.h | 7 ++++++- include/linux/mm.h | 16 ++++++++++++++++ 6 files changed, 31 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 2c29239d05c3..846c563689a8 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -96,7 +96,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, unsigned long addr) { - tlb_remove_ptdesc(tlb, virt_to_ptdesc(pudp)); + struct ptdesc *ptdesc = virt_to_ptdesc(pudp); + + pagetable_pud_dtor(ptdesc); + tlb_remove_ptdesc(tlb, ptdesc); } #endif diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h index 79470f0b4f1d..4e2d6b7ca2ee 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -84,6 +84,7 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) if (!ptdesc) return NULL; + pagetable_pud_ctor(ptdesc); pud = ptdesc_address(ptdesc); pud_init(pud); diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h index 40e40a7eb94a..f4440edcd8fe 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -95,6 +95,7 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) if (!ptdesc) return NULL; + pagetable_pud_ctor(ptdesc); pud = ptdesc_address(ptdesc); pud_init(pud); diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 9deadf517f14..0cbc1b8e8e3d 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -76,6 +76,9 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { + struct ptdesc *ptdesc = virt_to_ptdesc(pud); + + pagetable_pud_dtor(ptdesc); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(pud)); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index c75d4a753849..879e5f8aa5e9 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -169,6 +169,8 @@ static inline pud_t *__pud_alloc_one(struct mm_struct *mm, unsigned long addr) ptdesc = pagetable_alloc(gfp, 0); if (!ptdesc) return NULL; + + pagetable_pud_ctor(ptdesc); return ptdesc_address(ptdesc); } @@ -190,8 +192,11 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) static inline void __pud_free(struct mm_struct *mm, pud_t *pud) { + struct ptdesc *ptdesc = virt_to_ptdesc(pud); + BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - pagetable_free(virt_to_ptdesc(pud)); + pagetable_pud_dtor(ptdesc); + pagetable_free(ptdesc); } #ifndef __HAVE_ARCH_PUD_FREE diff --git a/include/linux/mm.h b/include/linux/mm.h index 12335de50140..2232bfebb88a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3049,6 +3049,22 @@ static inline spinlock_t *pud_lock(struct mm_struct *mm, pud_t *pud) return ptl; } +static inline void pagetable_pud_ctor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_set_pgtable(folio); + lruvec_stat_add_folio(folio, NR_PAGETABLE); +} + +static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + extern void __init pagecache_init(void); extern void free_initmem(void);