From patchwork Fri Jun 22 03:51:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 10481157 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 0BD8F60380 for ; Fri, 22 Jun 2018 03:55:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F099C28F91 for ; Fri, 22 Jun 2018 03:55:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E4EE228F98; Fri, 22 Jun 2018 03:55:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 271F028F91 for ; Fri, 22 Jun 2018 03:55:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 67DF96B0273; Thu, 21 Jun 2018 23:55:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 602BE6B0274; Thu, 21 Jun 2018 23:55:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4571C6B0275; Thu, 21 Jun 2018 23:55:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f71.google.com (mail-pl0-f71.google.com [209.85.160.71]) by kanga.kvack.org (Postfix) with ESMTP id EF45B6B0273 for ; Thu, 21 Jun 2018 23:55:42 -0400 (EDT) Received: by mail-pl0-f71.google.com with SMTP id 39-v6so2958119ple.6 for ; Thu, 21 Jun 2018 20:55:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=7iszdqyVdvPLnVNzia+VBmnnY1M+aYWs/fqckPQOWtY=; b=nI/AiMFFXY8WP+6WWLxQTacJ/DTNHjwT0Se3NBpFMERH4YWduiElIkysLxxQgy+BHK J+n4dwxiSzTCTodXN7xSXPEbOwq5NgP9B/oGnZwQw01LkfqKrcHigQDyqwBIxCir5t9J atfVQh3+4KY39iSrf9YSLUu4Tz9JzU32gzXKiTH832qYRGG/M2sBbhH/Bgrv5Fn3aX2X 34xSmbwdb9hn6ZJkNfJvWeH3lukyXxxxVTe4PZ/dz34g23qtueeVnmxsTjzAGlMyEU/g RvDYe0tp74Nc4lhW93CEHsSa3R5znnn3tarBoabVxnq4N8xKPp/rDhKM+QjTQejuRFt4 d2RA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APt69E1rBJ/1OwVCs038xVvhbO+Fu1t5W8WVK4JltGZiIygcP3dGY+MG NQ5F0MKdI4qz7E9mhchjN2W2Zt+GKoFqpT7BrvWwV323ahbPkGOhnOjZjQl2SUUlgJZRqffgICX IYaNAi5EbU6LRr72OfqX1Ti/u6Kfvdc6LeFwUFbzSTE35bsW//uwq5+l9GCAGC8tOeg== X-Received: by 2002:a65:665a:: with SMTP id z26-v6mr4021699pgv.117.1529639742615; Thu, 21 Jun 2018 20:55:42 -0700 (PDT) X-Google-Smtp-Source: ADUXVKLfjgxOJnpCiuuXLGlKv53vBynEMvd4FWAYBVHIGx/1KfQtKETN7xIuwNz+dRETGYno2lYa X-Received: by 2002:a65:665a:: with SMTP id z26-v6mr4021675pgv.117.1529639741823; Thu, 21 Jun 2018 20:55:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529639741; cv=none; d=google.com; s=arc-20160816; b=TiNRMc47aPzKXL7G7KkDZaAm2wUXpVGEkt73dzMRiuEzpBoH6i6MEynoaMmZRJgYwb liEzeSun82S9EEotKb8foo0klP+4ekByXmjA5dzwapo7ArFKNwDriCuIIVLYRXj0FxHz 5XP+bGlit1jwxnid6pf7csV7C9p+60fZb6tw4UotO+htIBQ1L4V3MF174p8kRAgSucru 6KcicxLV7ZKfVR9vjf57xgNNUzPh7wgC+mYwqJ93579XuxqxOOwLXT6u/Ut7EzMq9V+E DT+xC3oooQPWoxTNGfOBGJJJbgNgFO9nsz+pxnqOQjgoQth7N90Fss9USoEoPhv0pdee 0T5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=7iszdqyVdvPLnVNzia+VBmnnY1M+aYWs/fqckPQOWtY=; b=Gten9hqr4Cxegpme/wICJTDwdb/bcWi0pa6SME0BeGQtwNqkuUZJXmUGLNkPCUzaap 4EETKanCpGhymdlemlk1gM7OwMB3ShdxJ5HmKtWyE7J0iPuaAZhRDI31SUE/xIqhSuXT qUBnCueWdqsCsys16InYqmD21vULhX8erSZluJylrUvrrE0hY9XsGIISAjyNvH0O6tr1 NsXOgSz0lFwEvdoVq8ncRmvBnKcI7+ecvPot1W1BTQutqyFKn+tWxaqQxnZZIPMOv+Tf VdftomIe08PiBfV3nBqXwXNalvBVvd/phuEFLNRfnqXjT2SzRKtQEg8KpxdaXJEmRjUA YYgA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id n61-v6si6160066plb.256.2018.06.21.20.55.41 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Jun 2018 20:55:41 -0700 (PDT) Received-SPF: pass (google.com: domain of ying.huang@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Jun 2018 20:55:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,255,1526367600"; d="scan'208";a="65335155" Received: from wanpingl-mobl.ccr.corp.intel.com (HELO yhuang6-ux31a.ccr.corp.intel.com) ([10.254.212.200]) by fmsmga004.fm.intel.com with ESMTP; 21 Jun 2018 20:55:39 -0700 From: "Huang, Ying" To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , "Kirill A. Shutemov" , Andrea Arcangeli , Michal Hocko , Johannes Weiner , Shaohua Li , Hugh Dickins , Minchan Kim , Rik van Riel , Dave Hansen , Naoya Horiguchi , Zi Yan , Daniel Jordan Subject: [PATCH -mm -v4 12/21] mm, THP, swap: Support PMD swap mapping in swapoff Date: Fri, 22 Jun 2018 11:51:42 +0800 Message-Id: <20180622035151.6676-13-ying.huang@intel.com> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20180622035151.6676-1-ying.huang@intel.com> References: <20180622035151.6676-1-ying.huang@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Huang Ying During swapoff, for a huge swap cluster, we need to allocate a THP, read its contents into the THP and unuse the PMD and PTE swap mappings to it. If failed to allocate a THP, the huge swap cluster will be split. During unuse, if it is found that the swap cluster mapped by a PMD swap mapping is split already, we will split the PMD swap mapping and unuse the PTEs. Signed-off-by: "Huang, Ying" Cc: "Kirill A. Shutemov" Cc: Andrea Arcangeli Cc: Michal Hocko Cc: Johannes Weiner Cc: Shaohua Li Cc: Hugh Dickins Cc: Minchan Kim Cc: Rik van Riel Cc: Dave Hansen Cc: Naoya Horiguchi Cc: Zi Yan Cc: Daniel Jordan --- include/asm-generic/pgtable.h | 15 ++------ include/linux/huge_mm.h | 8 ++++ mm/huge_memory.c | 4 +- mm/swapfile.c | 86 ++++++++++++++++++++++++++++++++++++++++++- 4 files changed, 98 insertions(+), 15 deletions(-) diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index bb8354981a36..caa381962cd2 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -931,22 +931,13 @@ static inline int pmd_none_or_trans_huge_or_clear_bad(pmd_t *pmd) barrier(); #endif /* - * !pmd_present() checks for pmd migration entries - * - * The complete check uses is_pmd_migration_entry() in linux/swapops.h - * But using that requires moving current function and pmd_trans_unstable() - * to linux/swapops.h to resovle dependency, which is too much code move. - * - * !pmd_present() is equivalent to is_pmd_migration_entry() currently, - * because !pmd_present() pages can only be under migration not swapped - * out. - * - * pmd_none() is preseved for future condition checks on pmd migration + * pmd_none() is preseved for future condition checks on pmd swap * entries and not confusing with this function name, although it is * redundant with !pmd_present(). */ if (pmd_none(pmdval) || pmd_trans_huge(pmdval) || - (IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION) && !pmd_present(pmdval))) + ((IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION) || + IS_ENABLED(CONFIG_THP_SWAP)) && !pmd_present(pmdval))) return 1; if (unlikely(pmd_bad(pmdval))) { pmd_clear_bad(pmd); diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 7931fa888f11..bc92c2944756 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -406,6 +406,8 @@ static inline gfp_t alloc_hugepage_direct_gfpmask(struct vm_area_struct *vma) #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #ifdef CONFIG_THP_SWAP +extern int split_huge_swap_pmd(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long address, pmd_t orig_pmd); extern int do_huge_pmd_swap_page(struct vm_fault *vmf, pmd_t orig_pmd); static inline bool transparent_hugepage_swapin_enabled( @@ -431,6 +433,12 @@ static inline bool transparent_hugepage_swapin_enabled( return false; } #else /* CONFIG_THP_SWAP */ +static inline int split_huge_swap_pmd(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long address, pmd_t orig_pmd) +{ + return 0; +} + static inline int do_huge_pmd_swap_page(struct vm_fault *vmf, pmd_t orig_pmd) { return 0; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index da42d1cdc26a..73fc77633642 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1663,8 +1663,8 @@ static void __split_huge_swap_pmd(struct vm_area_struct *vma, pmd_populate(mm, pmd, pgtable); } -static int split_huge_swap_pmd(struct vm_area_struct *vma, pmd_t *pmd, - unsigned long address, pmd_t orig_pmd) +int split_huge_swap_pmd(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long address, pmd_t orig_pmd) { struct mm_struct *mm = vma->vm_mm; spinlock_t *ptl; diff --git a/mm/swapfile.c b/mm/swapfile.c index e1e43654407c..34e64f3570c3 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1933,6 +1933,11 @@ static inline int pte_same_as_swp(pte_t pte, pte_t swp_pte) return pte_same(pte_swp_clear_soft_dirty(pte), swp_pte); } +static inline int pmd_same_as_swp(pmd_t pmd, pmd_t swp_pmd) +{ + return pmd_same(pmd_swp_clear_soft_dirty(pmd), swp_pmd); +} + /* * No need to decide whether this PTE shares the swap entry with others, * just let do_wp_page work it out if a write is requested later - to @@ -1994,6 +1999,57 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, return ret; } +#ifdef CONFIG_THP_SWAP +static int unuse_pmd(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long addr, swp_entry_t entry, struct page *page) +{ + struct mem_cgroup *memcg; + struct swap_info_struct *si; + spinlock_t *ptl; + int ret = 1; + + if (mem_cgroup_try_charge(page, vma->vm_mm, GFP_KERNEL, + &memcg, true)) { + ret = -ENOMEM; + goto out_nolock; + } + + ptl = pmd_lock(vma->vm_mm, pmd); + if (unlikely(!pmd_same_as_swp(*pmd, swp_entry_to_pmd(entry)))) { + mem_cgroup_cancel_charge(page, memcg, true); + ret = 0; + goto out; + } + + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -HPAGE_PMD_NR); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR); + get_page(page); + set_pmd_at(vma->vm_mm, addr, pmd, + pmd_mkold(mk_huge_pmd(page, vma->vm_page_prot))); + page_add_anon_rmap(page, vma, addr, true); + mem_cgroup_commit_charge(page, memcg, true, true); + si = _swap_info_get(entry); + if (si) + swap_free_cluster(si, entry); + /* + * Move the page to the active list so it is not + * immediately swapped out again after swapon. + */ + activate_page(page); +out: + spin_unlock(ptl); +out_nolock: + return ret; +} +#else +static inline int unuse_pmd(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long addr, swp_entry_t entry, + struct page *page) +{ + return 0; +} +#endif + static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, swp_entry_t entry, struct page *page) @@ -2034,7 +2090,7 @@ static inline int unuse_pmd_range(struct vm_area_struct *vma, pud_t *pud, unsigned long addr, unsigned long end, swp_entry_t entry, struct page *page) { - pmd_t *pmd; + pmd_t swp_pmd = swp_entry_to_pmd(entry), *pmd, orig_pmd; unsigned long next; int ret; @@ -2042,6 +2098,24 @@ static inline int unuse_pmd_range(struct vm_area_struct *vma, pud_t *pud, do { cond_resched(); next = pmd_addr_end(addr, end); + orig_pmd = *pmd; + if (thp_swap_supported() && is_swap_pmd(orig_pmd)) { + if (likely(!pmd_same_as_swp(orig_pmd, swp_pmd))) + continue; + /* Huge cluster has been split already */ + if (!PageTransCompound(page)) { + ret = split_huge_swap_pmd(vma, pmd, + addr, orig_pmd); + if (ret) + return ret; + ret = unuse_pte_range(vma, pmd, addr, + next, entry, page); + } else + ret = unuse_pmd(vma, pmd, addr, entry, page); + if (ret) + return ret; + continue; + } if (pmd_none_or_trans_huge_or_clear_bad(pmd)) continue; ret = unuse_pte_range(vma, pmd, addr, next, entry, page); @@ -2206,6 +2280,7 @@ int try_to_unuse(unsigned int type, bool frontswap, * to prevent compiler doing * something odd. */ + struct swap_cluster_info *ci = NULL; unsigned char swcount; struct page *page; swp_entry_t entry; @@ -2235,6 +2310,7 @@ int try_to_unuse(unsigned int type, bool frontswap, * there are races when an instance of an entry might be missed. */ while ((i = find_next_to_unuse(si, i, frontswap)) != 0) { +retry: if (signal_pending(current)) { retval = -EINTR; break; @@ -2246,6 +2322,8 @@ int try_to_unuse(unsigned int type, bool frontswap, * page and read the swap into it. */ swap_map = &si->swap_map[i]; + if (si->cluster_info) + ci = si->cluster_info + i / SWAPFILE_CLUSTER; entry = swp_entry(type, i); page = read_swap_cache_async(entry, GFP_HIGHUSER_MOVABLE, NULL, 0, false); @@ -2266,6 +2344,12 @@ int try_to_unuse(unsigned int type, bool frontswap, */ if (!swcount || swcount == SWAP_MAP_BAD) continue; + /* Split huge cluster if failed to allocate huge page */ + if (thp_swap_supported() && cluster_is_huge(ci)) { + retval = split_swap_cluster(entry, false); + if (!retval || retval == -EEXIST) + goto retry; + } retval = -ENOMEM; break; }