From patchwork Wed May 9 08:38:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 10388653 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 2462A60170 for ; Wed, 9 May 2018 08:39:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 17CB928E24 for ; Wed, 9 May 2018 08:39:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0BFC728E29; Wed, 9 May 2018 08:39:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6029728E26 for ; Wed, 9 May 2018 08:39:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 941DB6B0499; Wed, 9 May 2018 04:39:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8C79E6B049B; Wed, 9 May 2018 04:39:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 73EB66B049C; Wed, 9 May 2018 04:39:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg0-f72.google.com (mail-pg0-f72.google.com [74.125.83.72]) by kanga.kvack.org (Postfix) with ESMTP id 354B26B0499 for ; Wed, 9 May 2018 04:39:26 -0400 (EDT) Received: by mail-pg0-f72.google.com with SMTP id f19-v6so19458492pgv.4 for ; Wed, 09 May 2018 01:39:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Lczodys+y4qcMMyafzab//BiB1YBFon9ry7mES9CfRs=; b=BioPZOl6hN36P+p2v9OHAfNk03gFiLQS9vtiChMKw71P7wWuxHEHeb4Ab4pdy8JkFe rmnDktQ6QHBa6AG3GRztWCV2r00+jqTIRK8KtrssV6ryLkcL5Kxy82gB2DSLEvrKdQGL Tc5/N+HAOOtE2169vmyBhQYbtZok67aLML7rm04woazpRBFLUV8K6v8CAZW98WiImqnO U357yVhDyU0veWLo525BBidi5hDyGL8jjycC4y/TxQrWzDpxwaB4U2S36itohy1CCOdj 03Tw46RduOJG7iwKuz/szf+qDLOG5I4xodyRUwJVtu86aXH/YkPecS9SfcQup0dttYzX 2sMA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com X-Gm-Message-State: ALQs6tAAFS3qSzxX8QXDHVDGRr1GpywpwnIyf9LbLmB5aKYosFzW7jcf birghmw4hFYDD02NyvjpkYM7W5lhNaxsbXIW5rzL71meppOYJrRqBn5gg8yHyriOhHwy+2tRoir DIZ8lRjr8Q8j3j+bGLPCTtCburTt4dykNVJv4Sh2I3FTbaEHChFs1CabXze+qW2/2Fg== X-Received: by 2002:a63:6fc8:: with SMTP id k191-v6mr35262669pgc.330.1525855165897; Wed, 09 May 2018 01:39:25 -0700 (PDT) X-Google-Smtp-Source: AB8JxZozY2WeQnT68kksRzLoPELkCrPpSIHZDBWmjNWTaKUxPSup0FHLYqs23VmPkeMHMLNrqD0I X-Received: by 2002:a63:6fc8:: with SMTP id k191-v6mr35262641pgc.330.1525855164986; Wed, 09 May 2018 01:39:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525855164; cv=none; d=google.com; s=arc-20160816; b=q1dmtMt8d0pYTUl15XMpZ/Uzb34OoS9yYC8UrcF9lO8aXvNCRM8pApwDpEt/peGlrb l5Frxw99m2ISNE8sMPKugOk1u57ZbTbefBD6vOBEsmcpDq7H0Y0PjTC2AtosqFhCRQDA 057mw6UKyUS78iIAhTH4swwYOD6mMHOm1B8asphWfgMb58Z67WEnTOGEmDefJinwp+CO CWTH1RfTAEurL/SzVrlxzL0cEL2hwFb9paeeX1TKh4WBSbdKTIDtC46VrQYt4WNxxQ8y 63bbpfZ5+xRZYO3Rq3E2EKtjWqkVrTkRSUDP7fLhVUm0VBEph0Q/sGSJk1pjf8vVhmnN xaSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=Lczodys+y4qcMMyafzab//BiB1YBFon9ry7mES9CfRs=; b=dqrceCQvOQ07OWRP+3qbYUJF9S0ptu3i7CkUWkvxG7wkq/J/6lGsorVsAP5KTH+4eD IfPoyrE58oPE2R6+mLlIA9EaFNhVbHUUaJyAUEdRe3cgC9wn+uKhepCBZ9uLKRUshrvo y7cE7zGNM9s1qrW1gdwjKK/t+jKcsjfpZ3jNAMlWq4S07ny4oimS/FuYrp+kvp0xq7dV AsD1v7jPXKOUh1nbnxYNpW7abECOI4cA8TQkdY/pqwpdvN5LY+UpH1Bhnb6mMhjPq/ay cfBi2b6hnXE8wDVMYwYnFryGbNxmhEwWe5K+yMhnIyElorF3ZWuRDKS4i6HFwkojJUe0 xC8A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com Received: from mga18.intel.com (mga18.intel.com. [134.134.136.126]) by mx.google.com with ESMTPS id y11-v6si18624861pgv.473.2018.05.09.01.39.24 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 May 2018 01:39:24 -0700 (PDT) Received-SPF: pass (google.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) client-ip=134.134.136.126; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga106.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 May 2018 01:39:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.49,381,1520924400"; d="scan'208";a="52769597" Received: from yhuang-gentoo.sh.intel.com ([10.239.193.148]) by fmsmga004.fm.intel.com with ESMTP; 09 May 2018 01:39:21 -0700 From: "Huang, Ying" To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , "Kirill A. Shutemov" , Andrea Arcangeli , Michal Hocko , Johannes Weiner , Shaohua Li , Hugh Dickins , Minchan Kim , Rik van Riel , Dave Hansen , Naoya Horiguchi , Zi Yan Subject: [PATCH -mm -V2 07/21] mm, THP, swap: Support PMD swap mapping in split_swap_cluster() Date: Wed, 9 May 2018 16:38:32 +0800 Message-Id: <20180509083846.14823-8-ying.huang@intel.com> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180509083846.14823-1-ying.huang@intel.com> References: <20180509083846.14823-1-ying.huang@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Huang Ying When splitting a THP in swap cache or failing to allocate a THP when swapin a huge swap cluster, the huge swap cluster will be split. In addition to clear the huge flag of the swap cluster, the PMD swap mapping count recorded in cluster_count() will be set to 0. But we will not touch PMD swap mappings themselves, because it is hard to find them all sometimes. When the PMD swap mappings are operated later, it will be found that the huge swap cluster has been split and the PMD swap mappings will be split at that time. Unless splitting a THP in swap cache (specified via "force" parameter), split_swap_cluster() will return -EEXIST if there is SWAP_HAS_CACHE flag in swap_map[offset]. Because this indicates there is a THP corresponds to this huge swap cluster, and it isn't desired to split the THP. When splitting a THP in swap cache, the position to call split_swap_cluster() is changed to before unlocking sub-pages. So that all sub-pages will be kept locked from the THP has been split to the huge swap cluster is split. This makes the code much easier to be reasoned. Signed-off-by: "Huang, Ying" Cc: "Kirill A. Shutemov" Cc: Andrea Arcangeli Cc: Michal Hocko Cc: Johannes Weiner Cc: Shaohua Li Cc: Hugh Dickins Cc: Minchan Kim Cc: Rik van Riel Cc: Dave Hansen Cc: Naoya Horiguchi Cc: Zi Yan --- include/linux/swap.h | 4 ++-- mm/huge_memory.c | 18 ++++++++++++------ mm/swapfile.c | 45 ++++++++++++++++++++++++++++++--------------- 3 files changed, 44 insertions(+), 23 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index bb9de2cb952a..878f132dabc0 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -617,10 +617,10 @@ static inline swp_entry_t get_swap_page(struct page *page) #endif /* CONFIG_SWAP */ #ifdef CONFIG_THP_SWAP -extern int split_swap_cluster(swp_entry_t entry); +extern int split_swap_cluster(swp_entry_t entry, bool force); extern int split_swap_cluster_map(swp_entry_t entry); #else -static inline int split_swap_cluster(swp_entry_t entry) +static inline int split_swap_cluster(swp_entry_t entry, bool force) { return 0; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 86800ef7c61c..fea9dcba7dc1 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2505,6 +2505,17 @@ static void __split_huge_page(struct page *page, struct list_head *list, unfreeze_page(head); + /* + * Split swap cluster before unlocking sub-pages. So all + * sub-pages will be kept locked from THP has been split to + * swap cluster is split. + */ + if (PageSwapCache(head)) { + swp_entry_t entry = { .val = page_private(head) }; + + split_swap_cluster(entry, true); + } + for (i = 0; i < HPAGE_PMD_NR; i++) { struct page *subpage = head + i; if (subpage == page) @@ -2731,12 +2742,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) __dec_node_page_state(page, NR_SHMEM_THPS); spin_unlock(&pgdata->split_queue_lock); __split_huge_page(page, list, flags); - if (PageSwapCache(head)) { - swp_entry_t entry = { .val = page_private(head) }; - - ret = split_swap_cluster(entry); - } else - ret = 0; + ret = 0; } else { if (IS_ENABLED(CONFIG_DEBUG_VM) && mapcount) { pr_alert("total_mapcount: %u, page_count(): %u\n", diff --git a/mm/swapfile.c b/mm/swapfile.c index acf2d0c30457..3316820cd3cd 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1414,21 +1414,6 @@ static void swapcache_free_cluster(swp_entry_t entry) } } } - -int split_swap_cluster(swp_entry_t entry) -{ - struct swap_info_struct *si; - struct swap_cluster_info *ci; - unsigned long offset = swp_offset(entry); - - si = _swap_info_get(entry); - if (!si) - return -EBUSY; - ci = lock_cluster(si, offset); - cluster_clear_huge(ci); - unlock_cluster(ci); - return 0; -} #else static inline void swapcache_free_cluster(swp_entry_t entry) { @@ -4067,6 +4052,36 @@ int split_swap_cluster_map(swp_entry_t entry) unlock_cluster(ci); return 0; } + +int split_swap_cluster(swp_entry_t entry, bool force) +{ + struct swap_info_struct *si; + struct swap_cluster_info *ci; + unsigned long offset = swp_offset(entry); + int ret = 0; + + si = get_swap_device(entry); + if (!si) + return -EINVAL; + ci = lock_cluster(si, offset); + /* The swap cluster has been split by someone else */ + if (!cluster_is_huge(ci)) + goto out; + VM_BUG_ON(!is_cluster_offset(offset)); + VM_BUG_ON(cluster_count(ci) < SWAPFILE_CLUSTER); + /* If not forced, don't split swap cluster has swap cache */ + if (!force && si->swap_map[offset] & SWAP_HAS_CACHE) { + ret = -EEXIST; + goto out; + } + cluster_set_count(ci, SWAPFILE_CLUSTER); + cluster_clear_huge(ci); + +out: + unlock_cluster(ci); + put_swap_device(si); + return ret; +} #endif static int __init swapfile_init(void)