From patchwork Sat Jul 31 06:39:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12412317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 930D4C4320A for ; Sat, 31 Jul 2021 06:39:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 22F9761058 for ; Sat, 31 Jul 2021 06:39:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 22F9761058 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 3C3D88D0001; Sat, 31 Jul 2021 02:39:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 34D096B0036; Sat, 31 Jul 2021 02:39:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1EF228D0001; Sat, 31 Jul 2021 02:39:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0103.hostedemail.com [216.40.44.103]) by kanga.kvack.org (Postfix) with ESMTP id F25126B0033 for ; Sat, 31 Jul 2021 02:39:45 -0400 (EDT) Received: from smtpin32.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 85921180ACF62 for ; Sat, 31 Jul 2021 06:39:45 +0000 (UTC) X-FDA: 78421932330.32.945D681 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf07.hostedemail.com (Postfix) with ESMTP id 334171007E56 for ; Sat, 31 Jul 2021 06:39:45 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id w200-20020a25c7d10000b02905585436b530so13054733ybe.21 for ; Fri, 30 Jul 2021 23:39:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=BM+FE+/TiUx2OjFlRwtIO4yCUv9GYAVqgfjFJ6CUhoU=; b=WsYJB5oepFO3Wnd6PD+UXgOWzwQ85U4gvNUBQPL78jtZrSF4sTHPbyiNtqUk23nQhz c1/PYP5BlzvA/wilIXGO7/qINCd35Q73p17mn2jO/lk5M7RAh5BmffbH+ZRzbwuLlOAF 8HjeTu+PZBemftmZiO7WaTDU6M4xV0XSWMl1BXM0uHxPkwGsk578T+Uz5CQFs9IxGrHm jTmnq7qqKeYJz/kOeyz5dSHQoXn9qlKmIq8em5IZo+2APiFd8KgFFxvciDyawbOIwJ87 LeRs/nPRsn22A18v740Ymk2mMfcVnytiNyiQK5P0kmVBL/s19JKAYoiLqpdZyt3QI6uJ ok6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=BM+FE+/TiUx2OjFlRwtIO4yCUv9GYAVqgfjFJ6CUhoU=; b=kWdXehJ+uaexFW9U7npdV3zJ83jutqg6WtaxojVzqitKqGEOFIaXgS0t8Ev0zXYvEp fQNKtWYD4hmjvmOnm5lnkIVTGpgTiaJXq0AZEHupJrp5JFxiv0B8aSdVSxRcMjiWxgS3 +L3IUdrA4mwm/oHdfNfhL+3Ip/1F1xlp0Xci/s1EgAn2Dc5P8RpfxCspQiPNHmBPzhp0 BL4ovBfvBL2IpdsuxUWHP7heAM2xZsr9FLWfvPCfYcSsD8FoW1OOND3JUpX+7Uh0DZ5a tAKfGvOHt4H34azmsC4NHJo75o2QZu1wshnmdtaUg1FqQDT+hBXeXfYuD5y3uoJMjFiv sobA== X-Gm-Message-State: AOAM530UUCPmmLSdfERFww1B8k1HoDCHq7gxpB8kqQz7bqtzXbARTLXV SPIhwgeTlOiN4CqffRUTed9Pa/IDpBxv38XbrygiguOe01L8R8LaEKWDKVDvtlZKV16QTokZHyu 15uc2JjVY0PnBd/vy/5rOBD2R3AgkTpEawNEM3hq5B7LyzB0fV3+HEbyD X-Google-Smtp-Source: ABdhPJxiboigXtft6Tow9VLWZOsDXlaor5MrVjIBKJWd0LwIdCmmJ7JyIMMmMlT45SzkfdIze2fknU1vGMg= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:2b92:c131:b08a:84c7]) (user=yuzhao job=sendgmr) by 2002:a25:bc0f:: with SMTP id i15mr8025092ybh.233.1627713583285; Fri, 30 Jul 2021 23:39:43 -0700 (PDT) Date: Sat, 31 Jul 2021 00:39:36 -0600 In-Reply-To: <20210731063938.1391602-1-yuzhao@google.com> Message-Id: <20210731063938.1391602-2-yuzhao@google.com> Mime-Version: 1.0 References: <20210731063938.1391602-1-yuzhao@google.com> X-Mailer: git-send-email 2.32.0.554.ge1b32706d8-goog Subject: [PATCH 1/3] mm: don't take lru lock when splitting isolated thp From: Yu Zhao To: linux-mm@kvack.org Cc: Andrew Morton , Hugh Dickins , "Kirill A . Shutemov" , Matthew Wilcox , Vlastimil Babka , Yang Shi , Zi Yan , linux-kernel@vger.kernel.org, Yu Zhao , Shuang Zhai X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 334171007E56 Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=WsYJB5oe; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3L_AEYQYKCDIminVOcUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3L_AEYQYKCDIminVOcUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yuzhao.bounces.google.com X-Stat-Signature: yagzj3bp5pdrk7adga6eb39cwxjcgkos X-HE-Tag: 1627713585-707244 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We won't put its tail pages on lru when splitting an isolated thp under reclaim or migration, and therefore we don't need to take the lru lock. Signed-off-by: Yu Zhao Tested-by: Shuang Zhai --- mm/huge_memory.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index afff3ac87067..d8b655856e79 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2342,17 +2342,14 @@ static void remap_page(struct page *page, unsigned int nr) } } -static void lru_add_page_tail(struct page *head, struct page *tail, - struct lruvec *lruvec, struct list_head *list) +static void lru_add_page_tail(struct page *head, struct page *tail, struct list_head *list) { VM_BUG_ON_PAGE(!PageHead(head), head); VM_BUG_ON_PAGE(PageCompound(tail), head); VM_BUG_ON_PAGE(PageLRU(tail), head); - lockdep_assert_held(&lruvec->lru_lock); if (list) { - /* page reclaim is reclaiming a huge page */ - VM_WARN_ON(PageLRU(head)); + /* page reclaim or migration is splitting an isolated thp */ get_page(tail); list_add_tail(&tail->lru, list); } else { @@ -2363,8 +2360,7 @@ static void lru_add_page_tail(struct page *head, struct page *tail, } } -static void __split_huge_page_tail(struct page *head, int tail, - struct lruvec *lruvec, struct list_head *list) +static void __split_huge_page_tail(struct page *head, int tail, struct list_head *list) { struct page *page_tail = head + tail; @@ -2425,19 +2421,21 @@ static void __split_huge_page_tail(struct page *head, int tail, * pages to show after the currently processed elements - e.g. * migrate_pages */ - lru_add_page_tail(head, page_tail, lruvec, list); + lru_add_page_tail(head, page_tail, list); } static void __split_huge_page(struct page *page, struct list_head *list, pgoff_t end) { struct page *head = compound_head(page); - struct lruvec *lruvec; + struct lruvec *lruvec = NULL; struct address_space *swap_cache = NULL; unsigned long offset = 0; unsigned int nr = thp_nr_pages(head); int i; + VM_BUG_ON_PAGE(list && PageLRU(head), head); + /* complete memcg works before add pages to LRU */ split_page_memcg(head, nr); @@ -2450,10 +2448,11 @@ static void __split_huge_page(struct page *page, struct list_head *list, } /* lock lru list/PageCompound, ref frozen by page_ref_freeze */ - lruvec = lock_page_lruvec(head); + if (!list) + lruvec = lock_page_lruvec(head); for (i = nr - 1; i >= 1; i--) { - __split_huge_page_tail(head, i, lruvec, list); + __split_huge_page_tail(head, i, list); /* Some pages can be beyond i_size: drop them from page cache */ if (head[i].index >= end) { ClearPageDirty(head + i); @@ -2471,7 +2470,8 @@ static void __split_huge_page(struct page *page, struct list_head *list, } ClearPageCompound(head); - unlock_page_lruvec(lruvec); + if (lruvec) + unlock_page_lruvec(lruvec); /* Caller disabled irqs, so they are still disabled here */ split_page_owner(head, nr); @@ -2645,6 +2645,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) VM_BUG_ON_PAGE(is_huge_zero_page(head), head); VM_BUG_ON_PAGE(!PageLocked(head), head); VM_BUG_ON_PAGE(!PageCompound(head), head); + VM_BUG_ON_PAGE(list && PageLRU(head), head); if (PageWriteback(head)) return -EBUSY; From patchwork Sat Jul 31 06:39:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12412319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62CC5C4338F for ; Sat, 31 Jul 2021 06:39:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 12F7B61052 for ; Sat, 31 Jul 2021 06:39:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 12F7B61052 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id BB2A46B0033; Sat, 31 Jul 2021 02:39:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B3BC48D0002; Sat, 31 Jul 2021 02:39:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A03326B005D; Sat, 31 Jul 2021 02:39:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0196.hostedemail.com [216.40.44.196]) by kanga.kvack.org (Postfix) with ESMTP id 839556B0033 for ; Sat, 31 Jul 2021 02:39:46 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 28A2F184EB for ; Sat, 31 Jul 2021 06:39:46 +0000 (UTC) X-FDA: 78421932372.19.6A0FC5C Received: from mail-qk1-f202.google.com (mail-qk1-f202.google.com [209.85.222.202]) by imf28.hostedemail.com (Postfix) with ESMTP id 70286900438D for ; Sat, 31 Jul 2021 06:39:45 +0000 (UTC) Received: by mail-qk1-f202.google.com with SMTP id s206-20020a3745d70000b02903b9207abc7bso6612462qka.4 for ; Fri, 30 Jul 2021 23:39:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=s0577KTBq5IlED9+d/p0HQ6fImhPZLxAV9d3Y4Mj78k=; b=JsVUi6Cshol7JqaiPbmWtb0jBltGUAsUiTVIAkPq4qgVoGDn1xiRYAtE4oBXOagapF /0GxHhToMBmydIM3KD3k7c2uP+4o8emDxZOiGctKHjaottgLMDVySrweoSc/w+nr7uUb RZGTrHqPjGUr+Wyc47IXMWwYVUgfr4F69xtVPMvjM9GI00Ksg6l28Er/PadABeWKSskr oo7hmuWX0afR/0Ic0x/VZllAoefoKxJ49eQd57XhVnaH5+hEiy64x3vMLwGO9GzRDzcz DYMrZQ255iWt6d5paa3o12LiMShT2eVbhTy5LZGP2Bh37a+PdOtLonW2cLC2rP9USxdO 9w4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=s0577KTBq5IlED9+d/p0HQ6fImhPZLxAV9d3Y4Mj78k=; b=rBCDWiGBrBz0RWs+DrgOKjuFnocBtTa8hqOYVsLYVimHhWeLh3TRVafK9FXI0Bu7Sj JUl5NGlN2fNioUsZhzJ0G7UfpcEZ7JtDhg8PgjjE0ZydqdOjHX88enSE2oKzY6s5gOVL Tn6QlKce7SqAMVgu0skmedpzCUKtmPFj2RsrBizmTcEndstBkiq++90TYDPAHSCb3QER NA7Ps/3FIyy4EzuQ0jnLNZrH1vwZ8fEWYLIbuo0dhApSsYIGdIzP0cMF2OlAtlbjzQxK VNKFpa2OtopjF2n6SCiYSSBeCTDsY0+E7a5xP0gJ1TYe/9uLwHLoat48EvOnTPqDuI9X SbwQ== X-Gm-Message-State: AOAM533WlzYA++Lj3/Dq94kT7hWOvvjVptF18Wgpht6trNqi7WyTMfb1 YzQwc4yig6ufmRBVDQSv1zd8UXQQsekvCG1hKXlYhoR8jNWM/vPXOq/c2kse1jxCRlSbroyrDWl jbgIhM17NNdVLJUb8jYwFl9IQ/g0JtUCAv8YbPWpui6xOHbjCRxee1BWf X-Google-Smtp-Source: ABdhPJwZP7YlPH3vs70t1yZZQWOCe9C003giGh5MAHM1tgng90zlEpAZNper6xgPE76HUjXVP2xdOMN7L1M= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:2b92:c131:b08a:84c7]) (user=yuzhao job=sendgmr) by 2002:a05:6214:146c:: with SMTP id c12mr6778672qvy.44.1627713584655; Fri, 30 Jul 2021 23:39:44 -0700 (PDT) Date: Sat, 31 Jul 2021 00:39:37 -0600 In-Reply-To: <20210731063938.1391602-1-yuzhao@google.com> Message-Id: <20210731063938.1391602-3-yuzhao@google.com> Mime-Version: 1.0 References: <20210731063938.1391602-1-yuzhao@google.com> X-Mailer: git-send-email 2.32.0.554.ge1b32706d8-goog Subject: [PATCH 2/3] mm: free zapped tail pages when splitting isolated thp From: Yu Zhao To: linux-mm@kvack.org Cc: Andrew Morton , Hugh Dickins , "Kirill A . Shutemov" , Matthew Wilcox , Vlastimil Babka , Yang Shi , Zi Yan , linux-kernel@vger.kernel.org, Yu Zhao , Shuang Zhai Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=JsVUi6Cs; spf=pass (imf28.hostedemail.com: domain of 3MPAEYQYKCDMnjoWPdVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yuzhao.bounces.google.com designates 209.85.222.202 as permitted sender) smtp.mailfrom=3MPAEYQYKCDMnjoWPdVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 70286900438D X-Stat-Signature: 97iraye7ijh9cnxkd8go8h5kbqh6zm1y X-HE-Tag: 1627713585-317544 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If a tail page has only two references left, one inherited from the isolation of its head and the other from lru_add_page_tail() which we are about to drop, it means this tail page was concurrently zapped. Then we can safely free it and save page reclaim or migration the trouble of trying it. Signed-off-by: Yu Zhao Tested-by: Shuang Zhai --- include/linux/vm_event_item.h | 1 + mm/huge_memory.c | 28 ++++++++++++++++++++++++++++ mm/vmstat.c | 1 + 3 files changed, 30 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index ae0dd1948c2b..829eeac84094 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -99,6 +99,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD THP_SPLIT_PUD, #endif + THP_SPLIT_FREE, THP_ZERO_PAGE_ALLOC, THP_ZERO_PAGE_ALLOC_FAILED, THP_SWPOUT, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d8b655856e79..5120478bca41 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2432,6 +2432,8 @@ static void __split_huge_page(struct page *page, struct list_head *list, struct address_space *swap_cache = NULL; unsigned long offset = 0; unsigned int nr = thp_nr_pages(head); + LIST_HEAD(pages_to_free); + int nr_pages_to_free = 0; int i; VM_BUG_ON_PAGE(list && PageLRU(head), head); @@ -2506,6 +2508,25 @@ static void __split_huge_page(struct page *page, struct list_head *list, continue; unlock_page(subpage); + /* + * If a tail page has only two references left, one inherited + * from the isolation of its head and the other from + * lru_add_page_tail() which we are about to drop, it means this + * tail page was concurrently zapped. Then we can safely free it + * and save page reclaim or migration the trouble of trying it. + */ + if (list && page_ref_freeze(subpage, 2)) { + VM_BUG_ON_PAGE(PageLRU(subpage), subpage); + VM_BUG_ON_PAGE(PageCompound(subpage), subpage); + VM_BUG_ON_PAGE(page_mapped(subpage), subpage); + + ClearPageActive(subpage); + ClearPageUnevictable(subpage); + list_move(&subpage->lru, &pages_to_free); + nr_pages_to_free++; + continue; + } + /* * Subpages may be freed if there wasn't any mapping * like if add_to_swap() is running on a lru page that @@ -2515,6 +2536,13 @@ static void __split_huge_page(struct page *page, struct list_head *list, */ put_page(subpage); } + + if (!nr_pages_to_free) + return; + + mem_cgroup_uncharge_list(&pages_to_free); + free_unref_page_list(&pages_to_free); + count_vm_events(THP_SPLIT_FREE, nr_pages_to_free); } int total_mapcount(struct page *page) diff --git a/mm/vmstat.c b/mm/vmstat.c index b0534e068166..f486e5d98d96 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1300,6 +1300,7 @@ const char * const vmstat_text[] = { #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD "thp_split_pud", #endif + "thp_split_free", "thp_zero_page_alloc", "thp_zero_page_alloc_failed", "thp_swpout", From patchwork Sat Jul 31 06:39:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12412321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A185CC432BE for ; Sat, 31 Jul 2021 06:39:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4B1E261042 for ; Sat, 31 Jul 2021 06:39:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 4B1E261042 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id CC71B8D0003; Sat, 31 Jul 2021 02:39:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C017C8D0002; Sat, 31 Jul 2021 02:39:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2BCA8D0003; Sat, 31 Jul 2021 02:39:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0001.hostedemail.com [216.40.44.1]) by kanga.kvack.org (Postfix) with ESMTP id 8029C8D0002 for ; Sat, 31 Jul 2021 02:39:47 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 36EEF8249980 for ; Sat, 31 Jul 2021 06:39:47 +0000 (UTC) X-FDA: 78421932414.23.1D4B017 Received: from mail-qv1-f74.google.com (mail-qv1-f74.google.com [209.85.219.74]) by imf26.hostedemail.com (Postfix) with ESMTP id A552A200FC89 for ; Sat, 31 Jul 2021 06:39:46 +0000 (UTC) Received: by mail-qv1-f74.google.com with SMTP id fq10-20020a056214258ab02903395e637cf9so4526172qvb.15 for ; Fri, 30 Jul 2021 23:39:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+MG7KO18FjRhg16Po5bX500FIuiDvVXfmihsBqeNPd4=; b=P8UbmNylYr5Sj7rFQJyfHVoeATceJcM9Y3LBhLLzjMS852Qe3H6SWfkt2O49fY3w64 SAs+YU79iJfg9I8SDTDk0mYD8I5uRxmUqM4e85Y5rvAfVddKAex0mEhQ5nOzEV0FvMzn wNS764jptVkqnrta+p2pHUNy37vYmIaU0nnL7UKVVe5faH8fjZwnnMI0wqNC7WOU08Cy mjkfygihnZwJ/1LdDtkQlr7bz67nznmfP4tQGNz+mOhlMzAdmL7TDx+/bl8WEZEmduKl RxtY2tp4N9pttsT0R7aj5O4VfDJPBOXvd9PCb7oIGtzZelh0HgNdo4uNkf+cLGlpr+q6 mUDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=+MG7KO18FjRhg16Po5bX500FIuiDvVXfmihsBqeNPd4=; b=XEeFNT1Po161qGKrULaD5o6YnWhTrGzaXB2ZEqjY0IThh8adj7LnAvZHVXSdNt1UuQ Ztb+BtprQ4SxnAHVIiicWHFCzEOBJZQTurw6gbdi4HB70NMV+RUc7OAQsj57ice+ON3C hxoev4hgheEKtnQ6S4Ou6bj0t2p/f1/R+Vj+rpaQM3ZJK3WwTVLbH5KADKEXR8+omuEV ZNoyxtxrcAB4hqg7XgB0cRp4lWap1ZujvD+nwceZ5quGfTWb0nE3M6EVxJp1qiJ6Uxza 1gvtdtEFGlVklZHuulu+x7iEUPuDsqyx2t9e1R0lpkY4vPSqj4/kJQbkuhmkLNJhdBuI LbRA== X-Gm-Message-State: AOAM532MJY1tIh8pz49wwEPyPvd1/RYZiZd4YdL8yY781UYaBxtjKj/b eGUf7cGSyz9IK2mVLDt9PjjIkZmnBUu7MuBGrIi+xltZ/U1z4Cc0n9Rc82WrAddO5MmopX9yRfE cUxRo/462Uh1inFYOzqw9cWIU1P8wPhK5j9gFNH92P00s2P82eKfJ0rxS X-Google-Smtp-Source: ABdhPJyfLw+JGHTDtkurgYLTls9LSts2YZqcj8OoZnlbt/VPXwZmBRMFEnJJe1sPVvCKW+sKmgcQEMx43eg= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:2b92:c131:b08a:84c7]) (user=yuzhao job=sendgmr) by 2002:ad4:59c6:: with SMTP id el6mr6675401qvb.61.1627713585980; Fri, 30 Jul 2021 23:39:45 -0700 (PDT) Date: Sat, 31 Jul 2021 00:39:38 -0600 In-Reply-To: <20210731063938.1391602-1-yuzhao@google.com> Message-Id: <20210731063938.1391602-4-yuzhao@google.com> Mime-Version: 1.0 References: <20210731063938.1391602-1-yuzhao@google.com> X-Mailer: git-send-email 2.32.0.554.ge1b32706d8-goog Subject: [PATCH 3/3] mm: don't remap clean subpages when splitting isolated thp From: Yu Zhao To: linux-mm@kvack.org Cc: Andrew Morton , Hugh Dickins , "Kirill A . Shutemov" , Matthew Wilcox , Vlastimil Babka , Yang Shi , Zi Yan , linux-kernel@vger.kernel.org, Yu Zhao , Shuang Zhai X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A552A200FC89 Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=P8UbmNyl; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3MfAEYQYKCDQokpXQeWeeWbU.SecbYdkn-ccalQSa.ehW@flex--yuzhao.bounces.google.com designates 209.85.219.74 as permitted sender) smtp.mailfrom=3MfAEYQYKCDQokpXQeWeeWbU.SecbYdkn-ccalQSa.ehW@flex--yuzhao.bounces.google.com X-Stat-Signature: acte3fpxgn1jrxnjcbc4dagjobwwcjww X-HE-Tag: 1627713586-831501 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Here being clean means containing only zeros and inaccessible to userspace. When splitting an isolated thp under reclaim or migration, there is no need to remap its clean subpages because they can be faulted in anew. Not remapping them avoids writeback or copying during reclaim or migration. This is particularly helpful when the internal fragmentation of a thp is high, i.e., it has many untouched subpages. Signed-off-by: Yu Zhao Tested-by: Shuang Zhai Reported-by: kernel test robot Reported-by: kernel test robot --- include/linux/rmap.h | 2 +- include/linux/vm_event_item.h | 1 + mm/huge_memory.c | 10 +++--- mm/migrate.c | 65 ++++++++++++++++++++++++++++++----- mm/vmstat.c | 1 + 5 files changed, 64 insertions(+), 15 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index c976cc6de257..4c2789d3fafb 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -243,7 +243,7 @@ int page_mkclean(struct page *); */ void page_mlock(struct page *page); -void remove_migration_ptes(struct page *old, struct page *new, bool locked); +void remove_migration_ptes(struct page *old, struct page *new, bool locked, bool unmap_clean); /* * Called by memory-failure.c to kill processes. diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 829eeac84094..a08fa70d8026 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -100,6 +100,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, THP_SPLIT_PUD, #endif THP_SPLIT_FREE, + THP_SPLIT_UNMAP, THP_ZERO_PAGE_ALLOC, THP_ZERO_PAGE_ALLOC_FAILED, THP_SWPOUT, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 5120478bca41..1653b84dc800 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2327,7 +2327,7 @@ static void unmap_page(struct page *page) VM_WARN_ON_ONCE_PAGE(page_mapped(page), page); } -static void remap_page(struct page *page, unsigned int nr) +static void remap_page(struct page *page, unsigned int nr, bool unmap_clean) { int i; @@ -2335,10 +2335,10 @@ static void remap_page(struct page *page, unsigned int nr) if (!PageAnon(page)) return; if (PageTransHuge(page)) { - remove_migration_ptes(page, page, true); + remove_migration_ptes(page, page, true, false); } else { for (i = 0; i < nr; i++) - remove_migration_ptes(page + i, page + i, true); + remove_migration_ptes(page + i, page + i, true, unmap_clean); } } @@ -2494,7 +2494,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, } local_irq_enable(); - remap_page(head, nr); + remap_page(head, nr, !!list); if (PageSwapCache(head)) { swp_entry_t entry = { .val = page_private(head) }; @@ -2769,7 +2769,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) if (mapping) xa_unlock(&mapping->i_pages); local_irq_enable(); - remap_page(head, thp_nr_pages(head)); + remap_page(head, thp_nr_pages(head), false); ret = -EBUSY; } diff --git a/mm/migrate.c b/mm/migrate.c index 7e240437e7d9..46c2a54c2ac1 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -168,14 +168,53 @@ void putback_movable_pages(struct list_head *l) } } +static bool try_to_unmap_clean(struct page_vma_mapped_walk *pvmw, struct page *page) +{ + void *addr; + bool dirty; + + VM_BUG_ON_PAGE(PageLRU(page), page); + VM_BUG_ON_PAGE(PageCompound(page), page); + VM_BUG_ON_PAGE(!PageAnon(page), page); + VM_BUG_ON_PAGE(!PageLocked(page), page); + VM_BUG_ON_PAGE(pte_present(*pvmw->pte), page); + + if (PageMlocked(page) || (pvmw->vma->vm_flags & VM_LOCKED)) + return false; + + /* + * The pmd entry mapping the old thp was flushed and the pte mapping + * this subpage has been non present. Therefore, this subpage is + * inaccessible. We don't need to remap it if it contains only zeros. + */ + addr = kmap_atomic(page); + dirty = !!memchr_inv(addr, 0, PAGE_SIZE); + kunmap_atomic(addr); + + if (dirty) + return false; + + pte_clear_not_present_full(pvmw->vma->vm_mm, pvmw->address, pvmw->pte, false); + dec_mm_counter(pvmw->vma->vm_mm, mm_counter(page)); + count_vm_event(THP_SPLIT_UNMAP); + + return true; +} + +struct rmap_walk_arg { + struct page *page; + bool unmap_clean; +}; + /* * Restore a potential migration pte to a working pte entry */ static bool remove_migration_pte(struct page *page, struct vm_area_struct *vma, - unsigned long addr, void *old) + unsigned long addr, void *arg) { + struct rmap_walk_arg *rmap_walk_arg = arg; struct page_vma_mapped_walk pvmw = { - .page = old, + .page = rmap_walk_arg->page, .vma = vma, .address = addr, .flags = PVMW_SYNC | PVMW_MIGRATION, @@ -200,6 +239,8 @@ static bool remove_migration_pte(struct page *page, struct vm_area_struct *vma, continue; } #endif + if (rmap_walk_arg->unmap_clean && try_to_unmap_clean(&pvmw, new)) + continue; get_page(new); pte = pte_mkold(mk_pte(new, READ_ONCE(vma->vm_page_prot))); @@ -267,13 +308,19 @@ static bool remove_migration_pte(struct page *page, struct vm_area_struct *vma, * Get rid of all migration entries and replace them by * references to the indicated page. */ -void remove_migration_ptes(struct page *old, struct page *new, bool locked) +void remove_migration_ptes(struct page *old, struct page *new, bool locked, bool unmap_clean) { + struct rmap_walk_arg rmap_walk_arg = { + .page = old, + .unmap_clean = unmap_clean, + }; struct rmap_walk_control rwc = { .rmap_one = remove_migration_pte, - .arg = old, + .arg = &rmap_walk_arg, }; + VM_BUG_ON_PAGE(unmap_clean && old != new, old); + if (locked) rmap_walk_locked(new, &rwc); else @@ -827,7 +874,7 @@ static int writeout(struct address_space *mapping, struct page *page) * At this point we know that the migration attempt cannot * be successful. */ - remove_migration_ptes(page, page, false); + remove_migration_ptes(page, page, false, false); rc = mapping->a_ops->writepage(page, &wbc); @@ -1070,7 +1117,7 @@ static int __unmap_and_move(struct page *page, struct page *newpage, if (page_was_mapped) remove_migration_ptes(page, - rc == MIGRATEPAGE_SUCCESS ? newpage : page, false); + rc == MIGRATEPAGE_SUCCESS ? newpage : page, false, false); out_unlock_both: unlock_page(newpage); @@ -1292,7 +1339,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, if (page_was_mapped) remove_migration_ptes(hpage, - rc == MIGRATEPAGE_SUCCESS ? new_hpage : hpage, false); + rc == MIGRATEPAGE_SUCCESS ? new_hpage : hpage, false, false); unlock_put_anon: unlock_page(new_hpage); @@ -2585,7 +2632,7 @@ static void migrate_vma_unmap(struct migrate_vma *migrate) if (!page || (migrate->src[i] & MIGRATE_PFN_MIGRATE)) continue; - remove_migration_ptes(page, page, false); + remove_migration_ptes(page, page, false, false); migrate->src[i] = 0; unlock_page(page); @@ -2963,7 +3010,7 @@ void migrate_vma_finalize(struct migrate_vma *migrate) newpage = page; } - remove_migration_ptes(page, newpage, false); + remove_migration_ptes(page, newpage, false, false); unlock_page(page); if (is_zone_device_page(page)) diff --git a/mm/vmstat.c b/mm/vmstat.c index f486e5d98d96..a83cbb6a4d70 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1301,6 +1301,7 @@ const char * const vmstat_text[] = { "thp_split_pud", #endif "thp_split_free", + "thp_split_unmap", "thp_zero_page_alloc", "thp_zero_page_alloc_failed", "thp_swpout",