From patchwork Mon Sep 28 17:54:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 11804451 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0F043618 for ; Mon, 28 Sep 2020 17:55:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A47CF208FE for ; Mon, 28 Sep 2020 17:55:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=sent.com header.i=@sent.com header.b="SFN+ToHp"; dkim=temperror (0-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="h4x5vkIt" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A47CF208FE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=sent.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1FD376B0087; Mon, 28 Sep 2020 13:55:25 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 16112900003; Mon, 28 Sep 2020 13:55:25 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 025E46B008A; Mon, 28 Sep 2020 13:55:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0241.hostedemail.com [216.40.44.241]) by kanga.kvack.org (Postfix) with ESMTP id E1C326B0087 for ; Mon, 28 Sep 2020 13:55:24 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 7EEF02471 for ; Mon, 28 Sep 2020 17:55:24 +0000 (UTC) X-FDA: 77313222168.07.land82_5a0538327183 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id 59C6A18062B7B for ; Mon, 28 Sep 2020 17:55:24 +0000 (UTC) X-Spam-Summary: 1,0,0,a687e31d75e281ca,d41d8cd98f00b204,zi.yan@sent.com,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1261:1311:1314:1345:1359:1437:1515:1535:1544:1711:1730:1747:1777:1792:1981:2194:2196:2199:2200:2393:2559:2562:2693:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3871:3874:4119:4250:4321:4385:4605:5007:6117:6119:6120:6261:6653:6742:7576:7903:7927:8784:9010:10004:11026:11473:11658:11914:12043:12216:12291:12296:12438:12555:12679:12895:12986:13161:13229:13255:13894:14096:14181:14721:21080:21451:21627:21990:30012:30054:30064,0,RBL:64.147.123.17:@sent.com:.lbl8.mailshell.net-64.100.201.100 62.18.0.100;04y8hr6zoodmhmegpzhocs6y4u5ciopbxstirxaoxup81sp16y8crgd1aqaibda.jb7qdeu6zc8kdk7hstynkjfgif6qgxuc5o1j3poa7hwtm6z5ebe6dndig1jiiyb.6-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: land82_5a0538327183 X-Filterd-Recvd-Size: 8731 Received: from wnew3-smtp.messagingengine.com (wnew3-smtp.messagingengine.com [64.147.123.17]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Mon, 28 Sep 2020 17:55:23 +0000 (UTC) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailnew.west.internal (Postfix) with ESMTP id 20126CA9; Mon, 28 Sep 2020 13:55:21 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Mon, 28 Sep 2020 13:55:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=from :to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; s=fm1; bh=Gw9wJp8IzBhsG H24yD81uyFcJHF+7lzeeR/XcfNfCU4=; b=SFN+ToHp6cDYhIHKZ6aqpXAKZhKxg JIKpwEvVd5BVBsllSuSC1ILsH9d1DXKq1Tdu+JcGFF0ChdtruiJoRjA6XVt3OTit J1B1qWqYMQVnVO2Capb3wOtBkCeKM81btetu2U2SKV9UStuM4YrNfVzMv4mKFYGG q6+pXjvoPVP2CtFoGvvZk51daMMV4iy7ubXuhU7PLHuCDiQUfFYuiSGYo2JMYOto TiGLpfsG6IYkfercEC3jvzIwQyWopKtPKW80ycBCN3qH65S70ZarTk178byTPjQW g0sZLftwy0nBdtiq+2suCJ7m97EgyyKqQblEu+F8oFSu3RLrnc6TZU7hg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:reply-to:subject :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=Gw9wJp8IzBhsGH24yD81uyFcJHF+7lzeeR/XcfNfCU4=; b=h4x5vkIt TzXUSQ5j3DQb3gs/I0aMZeDa5H7HGY1DL5RgahoQU8/AFWwORPr7liJJXkHNvPKE T02inzV9jWm/fM1GO/dgy+rfcQ7Col3DNVBa2mr9SZ85W8znEyxZ7NUO6Az+Yq2g VzevDyZPZDYFEtNOYQ88X1Bq4w/586jfPKW0qe87iApCwnELMiZVvlHZlI/mSmUj q3iJ0CqfqEnQ5lhtxyTQNr8Pz4u8MAK+HIUpT6b1XiyEw4oeAQ0YZdVCuTT5KVs8 Fcb1BnbcYo2iiLCU2STafCImHGpKcFmTAe6lMyP5PXR/rvWV8PH//pB/9vu00KTg P5GMaF6bHRitCQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedrvdeigdeliecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhhrggfgsedtkeertdertddtnecuhfhrohhmpegkihcujggr nhcuoeiiihdrhigrnhesshgvnhhtrdgtohhmqeenucggtffrrghtthgvrhhnpeduhfffve ektdduhfdutdfgtdekkedvhfetuedufedtgffgvdevleehheevjefgtdenucfkphepuddv rdegiedruddtiedrudeigeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmh grihhlfhhrohhmpeiiihdrhigrnhesshgvnhhtrdgtohhm X-ME-Proxy: Received: from nvrsysarch6.NVidia.COM (unknown [12.46.106.164]) by mail.messagingengine.com (Postfix) with ESMTPA id AC2BB306468A; Mon, 28 Sep 2020 13:55:19 -0400 (EDT) From: Zi Yan To: linux-mm@kvack.org Cc: "Kirill A . Shutemov" , Roman Gushchin , Rik van Riel , Matthew Wilcox , Shakeel Butt , Yang Shi , Jason Gunthorpe , Mike Kravetz , Michal Hocko , David Hildenbrand , William Kucharski , Andrea Arcangeli , John Hubbard , David Nellans , linux-kernel@vger.kernel.org, Zi Yan Subject: [RFC PATCH v2 05/30] mm: thp: add page table deposit/withdraw functions for PUD THP. Date: Mon, 28 Sep 2020 13:54:03 -0400 Message-Id: <20200928175428.4110504-6-zi.yan@sent.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200928175428.4110504-1-zi.yan@sent.com> References: <20200928175428.4110504-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zi Yan We deposit 512 PMD pages, each of which has 512 PTE pages deposited in its ->deposit_head, to mm->deposit_head_pud. They will be withdrawn and used when a PUD THP split into 512 PMD THPs. In this way, when any of the 512 PMD THPs is split further, we will use the existing code path to withdraw PTE pages for use. Signed-off-by: Zi Yan --- include/linux/mm.h | 2 ++ include/linux/mm_types.h | 3 +++ include/linux/pgtable.h | 3 +++ kernel/fork.c | 6 ++++++ mm/pgtable-generic.c | 23 +++++++++++++++++++++++ 5 files changed, 37 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 01b62da34794..8f54f06c8eb6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2321,6 +2321,8 @@ static inline spinlock_t *pud_lock(struct mm_struct *mm, pud_t *pud) return ptl; } +#define huge_pud_deposit_head(mm, pud) ((mm)->deposit_head_pud) + extern void __init pagecache_init(void); extern void __init free_area_init_memoryless_node(int nid); extern void free_initmem(void); diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index be842926577a..5ff4dd6a3e32 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -515,6 +515,9 @@ struct mm_struct { /* pgtable deposit list head, protected by page_table_lock */ struct llist_head deposit_head_pmd; #endif +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD + struct llist_head deposit_head_pud; /* protected by page_table_lock */ +#endif #ifdef CONFIG_NUMA_BALANCING /* * numa_next_scan is the next time that the PTEs will be marked diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 177eab8e1c31..1f6d46465c54 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -465,10 +465,13 @@ static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma, #ifndef __HAVE_ARCH_PGTABLE_DEPOSIT extern void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, pgtable_t pgtable); +extern void pgtable_trans_huge_pud_deposit(struct mm_struct *mm, pud_t *pudp, + pgtable_t pgtable); #endif #ifndef __HAVE_ARCH_PGTABLE_WITHDRAW extern pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp); +extern pgtable_t pgtable_trans_huge_pud_withdraw(struct mm_struct *mm, pud_t *pudp); #endif #ifdef CONFIG_TRANSPARENT_HUGEPAGE diff --git a/kernel/fork.c b/kernel/fork.c index 9c8e880538de..86fbeec751ef 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -663,6 +663,9 @@ static void check_mm(struct mm_struct *mm) #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS VM_BUG_ON_MM(!llist_empty(&mm->deposit_head_pmd), mm); #endif +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD + VM_BUG_ON_MM(!llist_empty(&mm->deposit_head_pud), mm); +#endif } #define allocate_mm() (kmem_cache_alloc(mm_cachep, GFP_KERNEL)) @@ -1023,6 +1026,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, init_tlb_flush_pending(mm); #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS init_llist_head(&mm->deposit_head_pmd); +#endif +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD + init_llist_head(&mm->deposit_head_pud); #endif mm_init_uprobes_state(mm); diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index dbb0154165f1..a014cf847067 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -166,6 +166,15 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, /* FIFO */ llist_add(&pgtable->deposit_node, &huge_pmd_deposit_head(mm, pmdp)); } + +void pgtable_trans_huge_pud_deposit(struct mm_struct *mm, pud_t *pudp, + pgtable_t pgtable) +{ + assert_spin_locked(pud_lockptr(mm, pudp)); + + /* FIFO */ + llist_add(&pgtable->deposit_node, &huge_pud_deposit_head(mm, pudp)); +} #endif #ifndef __HAVE_ARCH_PGTABLE_WITHDRAW @@ -183,6 +192,20 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp) struct page, deposit_node); return pgtable; } + +pgtable_t pgtable_trans_huge_pud_withdraw(struct mm_struct *mm, pud_t *pudp) +{ + pgtable_t pgtable; + + assert_spin_locked(pud_lockptr(mm, pudp)); + + /* only withdraw from a non empty list */ + VM_BUG_ON(llist_empty(&huge_pud_deposit_head(mm, pudp))); + /* FIFO */ + pgtable = llist_entry(llist_del_first(&huge_pud_deposit_head(mm, pmdp)), + struct page, deposit_node); + return pgtable; +} #endif #ifndef __HAVE_ARCH_PMDP_INVALIDATE