From patchwork Sat Jul 31 06:39:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12412319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62CC5C4338F for ; Sat, 31 Jul 2021 06:39:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 12F7B61052 for ; Sat, 31 Jul 2021 06:39:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 12F7B61052 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id BB2A46B0033; Sat, 31 Jul 2021 02:39:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B3BC48D0002; Sat, 31 Jul 2021 02:39:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A03326B005D; Sat, 31 Jul 2021 02:39:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0196.hostedemail.com [216.40.44.196]) by kanga.kvack.org (Postfix) with ESMTP id 839556B0033 for ; Sat, 31 Jul 2021 02:39:46 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 28A2F184EB for ; Sat, 31 Jul 2021 06:39:46 +0000 (UTC) X-FDA: 78421932372.19.6A0FC5C Received: from mail-qk1-f202.google.com (mail-qk1-f202.google.com [209.85.222.202]) by imf28.hostedemail.com (Postfix) with ESMTP id 70286900438D for ; Sat, 31 Jul 2021 06:39:45 +0000 (UTC) Received: by mail-qk1-f202.google.com with SMTP id s206-20020a3745d70000b02903b9207abc7bso6612462qka.4 for ; Fri, 30 Jul 2021 23:39:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=s0577KTBq5IlED9+d/p0HQ6fImhPZLxAV9d3Y4Mj78k=; b=JsVUi6Cshol7JqaiPbmWtb0jBltGUAsUiTVIAkPq4qgVoGDn1xiRYAtE4oBXOagapF /0GxHhToMBmydIM3KD3k7c2uP+4o8emDxZOiGctKHjaottgLMDVySrweoSc/w+nr7uUb RZGTrHqPjGUr+Wyc47IXMWwYVUgfr4F69xtVPMvjM9GI00Ksg6l28Er/PadABeWKSskr oo7hmuWX0afR/0Ic0x/VZllAoefoKxJ49eQd57XhVnaH5+hEiy64x3vMLwGO9GzRDzcz DYMrZQ255iWt6d5paa3o12LiMShT2eVbhTy5LZGP2Bh37a+PdOtLonW2cLC2rP9USxdO 9w4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=s0577KTBq5IlED9+d/p0HQ6fImhPZLxAV9d3Y4Mj78k=; b=rBCDWiGBrBz0RWs+DrgOKjuFnocBtTa8hqOYVsLYVimHhWeLh3TRVafK9FXI0Bu7Sj JUl5NGlN2fNioUsZhzJ0G7UfpcEZ7JtDhg8PgjjE0ZydqdOjHX88enSE2oKzY6s5gOVL Tn6QlKce7SqAMVgu0skmedpzCUKtmPFj2RsrBizmTcEndstBkiq++90TYDPAHSCb3QER NA7Ps/3FIyy4EzuQ0jnLNZrH1vwZ8fEWYLIbuo0dhApSsYIGdIzP0cMF2OlAtlbjzQxK VNKFpa2OtopjF2n6SCiYSSBeCTDsY0+E7a5xP0gJ1TYe/9uLwHLoat48EvOnTPqDuI9X SbwQ== X-Gm-Message-State: AOAM533WlzYA++Lj3/Dq94kT7hWOvvjVptF18Wgpht6trNqi7WyTMfb1 YzQwc4yig6ufmRBVDQSv1zd8UXQQsekvCG1hKXlYhoR8jNWM/vPXOq/c2kse1jxCRlSbroyrDWl jbgIhM17NNdVLJUb8jYwFl9IQ/g0JtUCAv8YbPWpui6xOHbjCRxee1BWf X-Google-Smtp-Source: ABdhPJwZP7YlPH3vs70t1yZZQWOCe9C003giGh5MAHM1tgng90zlEpAZNper6xgPE76HUjXVP2xdOMN7L1M= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:2b92:c131:b08a:84c7]) (user=yuzhao job=sendgmr) by 2002:a05:6214:146c:: with SMTP id c12mr6778672qvy.44.1627713584655; Fri, 30 Jul 2021 23:39:44 -0700 (PDT) Date: Sat, 31 Jul 2021 00:39:37 -0600 In-Reply-To: <20210731063938.1391602-1-yuzhao@google.com> Message-Id: <20210731063938.1391602-3-yuzhao@google.com> Mime-Version: 1.0 References: <20210731063938.1391602-1-yuzhao@google.com> X-Mailer: git-send-email 2.32.0.554.ge1b32706d8-goog Subject: [PATCH 2/3] mm: free zapped tail pages when splitting isolated thp From: Yu Zhao To: linux-mm@kvack.org Cc: Andrew Morton , Hugh Dickins , "Kirill A . Shutemov" , Matthew Wilcox , Vlastimil Babka , Yang Shi , Zi Yan , linux-kernel@vger.kernel.org, Yu Zhao , Shuang Zhai Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=JsVUi6Cs; spf=pass (imf28.hostedemail.com: domain of 3MPAEYQYKCDMnjoWPdVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yuzhao.bounces.google.com designates 209.85.222.202 as permitted sender) smtp.mailfrom=3MPAEYQYKCDMnjoWPdVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 70286900438D X-Stat-Signature: 97iraye7ijh9cnxkd8go8h5kbqh6zm1y X-HE-Tag: 1627713585-317544 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If a tail page has only two references left, one inherited from the isolation of its head and the other from lru_add_page_tail() which we are about to drop, it means this tail page was concurrently zapped. Then we can safely free it and save page reclaim or migration the trouble of trying it. Signed-off-by: Yu Zhao Tested-by: Shuang Zhai --- include/linux/vm_event_item.h | 1 + mm/huge_memory.c | 28 ++++++++++++++++++++++++++++ mm/vmstat.c | 1 + 3 files changed, 30 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index ae0dd1948c2b..829eeac84094 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -99,6 +99,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD THP_SPLIT_PUD, #endif + THP_SPLIT_FREE, THP_ZERO_PAGE_ALLOC, THP_ZERO_PAGE_ALLOC_FAILED, THP_SWPOUT, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d8b655856e79..5120478bca41 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2432,6 +2432,8 @@ static void __split_huge_page(struct page *page, struct list_head *list, struct address_space *swap_cache = NULL; unsigned long offset = 0; unsigned int nr = thp_nr_pages(head); + LIST_HEAD(pages_to_free); + int nr_pages_to_free = 0; int i; VM_BUG_ON_PAGE(list && PageLRU(head), head); @@ -2506,6 +2508,25 @@ static void __split_huge_page(struct page *page, struct list_head *list, continue; unlock_page(subpage); + /* + * If a tail page has only two references left, one inherited + * from the isolation of its head and the other from + * lru_add_page_tail() which we are about to drop, it means this + * tail page was concurrently zapped. Then we can safely free it + * and save page reclaim or migration the trouble of trying it. + */ + if (list && page_ref_freeze(subpage, 2)) { + VM_BUG_ON_PAGE(PageLRU(subpage), subpage); + VM_BUG_ON_PAGE(PageCompound(subpage), subpage); + VM_BUG_ON_PAGE(page_mapped(subpage), subpage); + + ClearPageActive(subpage); + ClearPageUnevictable(subpage); + list_move(&subpage->lru, &pages_to_free); + nr_pages_to_free++; + continue; + } + /* * Subpages may be freed if there wasn't any mapping * like if add_to_swap() is running on a lru page that @@ -2515,6 +2536,13 @@ static void __split_huge_page(struct page *page, struct list_head *list, */ put_page(subpage); } + + if (!nr_pages_to_free) + return; + + mem_cgroup_uncharge_list(&pages_to_free); + free_unref_page_list(&pages_to_free); + count_vm_events(THP_SPLIT_FREE, nr_pages_to_free); } int total_mapcount(struct page *page) diff --git a/mm/vmstat.c b/mm/vmstat.c index b0534e068166..f486e5d98d96 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1300,6 +1300,7 @@ const char * const vmstat_text[] = { #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD "thp_split_pud", #endif + "thp_split_free", "thp_zero_page_alloc", "thp_zero_page_alloc_failed", "thp_swpout",