From patchwork Fri Feb 28 03:38:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 11411463 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2B36E92A for ; Fri, 28 Feb 2020 03:38:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0202E24690 for ; Fri, 28 Feb 2020 03:38:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0202E24690 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EDEC06B0006; Thu, 27 Feb 2020 22:38:41 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E681A6B0007; Thu, 27 Feb 2020 22:38:41 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D2E516B0008; Thu, 27 Feb 2020 22:38:41 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0006.hostedemail.com [216.40.44.6]) by kanga.kvack.org (Postfix) with ESMTP id B9D056B0006 for ; Thu, 27 Feb 2020 22:38:41 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 8A6B9181AEF1A for ; Fri, 28 Feb 2020 03:38:41 +0000 (UTC) X-FDA: 76538128842.25.point12_11960d4f70f4c X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,ying.huang@intel.com,,RULES_HIT:30012:30054:30056:30064:30069:30070,0,RBL:192.55.52.151:@intel.com:.lbl8.mailshell.net-62.18.0.100 64.95.201.95,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: point12_11960d4f70f4c X-Filterd-Recvd-Size: 4094 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Fri, 28 Feb 2020 03:38:40 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 27 Feb 2020 19:38:40 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,493,1574150400"; d="scan'208";a="232107356" Received: from yhuang-dev.sh.intel.com ([10.239.159.23]) by orsmga008.jf.intel.com with ESMTP; 27 Feb 2020 19:38:37 -0800 From: "Huang, Ying" To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Dave Hansen , David Hildenbrand , Mel Gorman , Vlastimil Babka , Zi Yan , Michal Hocko , Minchan Kim , Johannes Weiner , Hugh Dickins Subject: [RFC 1/3] mm, migrate: Check return value of try_to_unmap() Date: Fri, 28 Feb 2020 11:38:17 +0800 Message-Id: <20200228033819.3857058-2-ying.huang@intel.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200228033819.3857058-1-ying.huang@intel.com> References: <20200228033819.3857058-1-ying.huang@intel.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Huang Ying During the page migration, try_to_unmap() is called to replace the page table entries with the migration entries. Now its return value is ignored, that is generally OK in most cases. But in theory, it's possible that try_to_unmap() return false (failure) for the page migration after arch_unmap_one() is called in unmap code. Even if without arch_unmap_one(), it makes code more robust for the future code changing to check the return value. Known issue: I don't know what is the appropriate error code for try_to_unmap() failure. Whether EIO is OK? Signed-off-by: "Huang, Ying" Cc: Dave Hansen Cc: David Hildenbrand Cc: Mel Gorman Cc: Vlastimil Babka Cc: Zi Yan Cc: Michal Hocko Cc: Minchan Kim Cc: Johannes Weiner Cc: Hugh Dickins --- mm/migrate.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 3900044cfaa6..981f8374a6ef 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1116,8 +1116,11 @@ static int __unmap_and_move(struct page *page, struct page *newpage, /* Establish migration ptes */ VM_BUG_ON_PAGE(PageAnon(page) && !PageKsm(page) && !anon_vma, page); - try_to_unmap(page, - TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS); + if (!try_to_unmap(page, + TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS)) { + rc = -EIO; + goto out_unlock_both; + } page_was_mapped = 1; } @@ -1337,8 +1340,11 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, goto put_anon; if (page_mapped(hpage)) { - try_to_unmap(hpage, - TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS); + if (!try_to_unmap(hpage, + TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS)) { + rc = -EIO; + goto unlock_both; + } page_was_mapped = 1; } @@ -1349,6 +1355,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, remove_migration_ptes(hpage, rc == MIGRATEPAGE_SUCCESS ? new_hpage : hpage, false); +unlock_both: unlock_page(new_hpage); put_anon: @@ -2558,8 +2565,7 @@ static void migrate_vma_unmap(struct migrate_vma *migrate) continue; if (page_mapped(page)) { - try_to_unmap(page, flags); - if (page_mapped(page)) + if (!try_to_unmap(page, flags)) goto restore; } From patchwork Fri Feb 28 03:38:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 11411465 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CE79B14E3 for ; Fri, 28 Feb 2020 03:38:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9B7E124690 for ; Fri, 28 Feb 2020 03:38:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9B7E124690 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BF7A36B0007; Thu, 27 Feb 2020 22:38:45 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BCFD36B0008; Thu, 27 Feb 2020 22:38:45 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ABD5C6B000A; Thu, 27 Feb 2020 22:38:45 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0081.hostedemail.com [216.40.44.81]) by kanga.kvack.org (Postfix) with ESMTP id 93D156B0007 for ; Thu, 27 Feb 2020 22:38:45 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 69CF12488 for ; Fri, 28 Feb 2020 03:38:45 +0000 (UTC) X-FDA: 76538129010.29.tree95_12258d3c6dc51 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,ying.huang@intel.com,,RULES_HIT:30003:30012:30025:30054:30064:30070,0,RBL:192.55.52.151:@intel.com:.lbl8.mailshell.net-62.18.0.100 64.95.201.95,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: tree95_12258d3c6dc51 X-Filterd-Recvd-Size: 7466 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Fri, 28 Feb 2020 03:38:44 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 27 Feb 2020 19:38:43 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,493,1574150400"; d="scan'208";a="232107370" Received: from yhuang-dev.sh.intel.com ([10.239.159.23]) by orsmga008.jf.intel.com with ESMTP; 27 Feb 2020 19:38:40 -0800 From: "Huang, Ying" To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Dave Hansen , David Hildenbrand , Mel Gorman , Vlastimil Babka , Zi Yan , Michal Hocko , Peter Zijlstra , Minchan Kim , Johannes Weiner , Hugh Dickins Subject: [RFC 2/3] mm: Add a new page flag PageLayzyFree() for MADV_FREE Date: Fri, 28 Feb 2020 11:38:18 +0800 Message-Id: <20200228033819.3857058-3-ying.huang@intel.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200228033819.3857058-1-ying.huang@intel.com> References: <20200228033819.3857058-1-ying.huang@intel.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Huang Ying Now !PageSwapBacked() is used as the flag for the pages freed lazily via MADV_FREE. This isn't obvious enough. So Dave suggested to add a new page flag for that to improve the code readability. Signed-off-by: "Huang, Ying" Suggested-by: Dave Hansen Cc: David Hildenbrand Cc: Mel Gorman Cc: Vlastimil Babka Cc: Zi Yan Cc: Michal Hocko Cc: Peter Zijlstra Cc: Minchan Kim Cc: Johannes Weiner Cc: Hugh Dickins --- include/linux/page-flags.h | 25 +++++++++++++++++++++++++ mm/rmap.c | 6 +++--- mm/swap.c | 11 +++-------- mm/vmscan.c | 7 +++---- 4 files changed, 34 insertions(+), 15 deletions(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 49c2697046b9..759748fbcfad 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -498,6 +498,31 @@ static __always_inline int PageKsm(struct page *page) TESTPAGEFLAG_FALSE(Ksm) #endif +/* + * For pages freed lazily via MADV_FREE. lazyfree pages are clean + * anonymous pages. They have SwapBacked flag cleared to distinguish + * with normal anonymous pages + */ +static __always_inline int PageLazyFree(struct page *page) +{ + page = compound_head(page); + return PageAnon(page) && !PageSwapBacked(page); +} + +static __always_inline void SetPageLazyFree(struct page *page) +{ + VM_BUG_ON_PAGE(PageTail(page), page); + VM_BUG_ON_PAGE(!PageAnon(page), page); + ClearPageSwapBacked(page); +} + +static __always_inline void ClearPageLazyFree(struct page *page) +{ + VM_BUG_ON_PAGE(PageTail(page), page); + VM_BUG_ON_PAGE(!PageAnon(page), page); + SetPageSwapBacked(page); +} + u64 stable_page_flags(struct page *page); static inline int PageUptodate(struct page *page) diff --git a/mm/rmap.c b/mm/rmap.c index 03c5b116d30e..1dcbb1771dd7 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1598,7 +1598,7 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, * Store the swap location in the pte. * See handle_pte_fault() ... */ - if (unlikely(PageSwapBacked(page) != PageSwapCache(page))) { + if (unlikely(PageLazyFree(page) == PageSwapCache(page))) { WARN_ON_ONCE(1); ret = false; /* We have to invalidate as we cleared the pte */ @@ -1609,7 +1609,7 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, } /* MADV_FREE page check */ - if (!PageSwapBacked(page)) { + if (PageLazyFree(page)) { if (!PageDirty(page)) { /* Invalidate as we cleared the pte */ mmu_notifier_invalidate_range(mm, @@ -1623,7 +1623,7 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, * discarded. Remap the page to page table. */ set_pte_at(mm, address, pvmw.pte, pteval); - SetPageSwapBacked(page); + ClearPageLazyFree(page); ret = false; page_vma_mapped_walk_done(&pvmw); break; diff --git a/mm/swap.c b/mm/swap.c index f502a2155e85..bd5e40e14c94 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -564,7 +564,7 @@ static void lru_deactivate_fn(struct page *page, struct lruvec *lruvec, static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec, void *arg) { - if (PageLRU(page) && PageAnon(page) && PageSwapBacked(page) && + if (PageLRU(page) && PageAnon(page) && !PageLazyFree(page) && !PageSwapCache(page) && !PageUnevictable(page)) { bool active = PageActive(page); @@ -572,12 +572,7 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec, LRU_INACTIVE_ANON + active); ClearPageActive(page); ClearPageReferenced(page); - /* - * lazyfree pages are clean anonymous pages. They have - * SwapBacked flag cleared to distinguish normal anonymous - * pages - */ - ClearPageSwapBacked(page); + SetPageLazyFree(page); add_page_to_lru_list(page, lruvec, LRU_INACTIVE_FILE); __count_vm_events(PGLAZYFREE, hpage_nr_pages(page)); @@ -678,7 +673,7 @@ void deactivate_page(struct page *page) */ void mark_page_lazyfree(struct page *page) { - if (PageLRU(page) && PageAnon(page) && PageSwapBacked(page) && + if (PageLRU(page) && PageAnon(page) && !PageLazyFree(page) && !PageSwapCache(page) && !PageUnevictable(page)) { struct pagevec *pvec = &get_cpu_var(lru_lazyfree_pvecs); diff --git a/mm/vmscan.c b/mm/vmscan.c index f14c8c6069a6..0aaee7052fb0 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1043,8 +1043,7 @@ static void page_check_dirty_writeback(struct page *page, * Anonymous pages are not handled by flushers and must be written * from reclaim context. Do not stall reclaim based on them */ - if (!page_is_file_cache(page) || - (PageAnon(page) && !PageSwapBacked(page))) { + if (!page_is_file_cache(page) || PageLazyFree(page)) { *dirty = false; *writeback = false; return; @@ -1235,7 +1234,7 @@ static unsigned long shrink_page_list(struct list_head *page_list, * Try to allocate it some swap space here. * Lazyfree page could be freed directly */ - if (PageAnon(page) && PageSwapBacked(page)) { + if (PageAnon(page) && !PageLazyFree(page)) { if (!PageSwapCache(page)) { if (!(sc->gfp_mask & __GFP_IO)) goto keep_locked; @@ -1411,7 +1410,7 @@ static unsigned long shrink_page_list(struct list_head *page_list, } } - if (PageAnon(page) && !PageSwapBacked(page)) { + if (PageLazyFree(page)) { /* follow __remove_mapping for reference */ if (!page_ref_freeze(page, 1)) goto keep_locked; From patchwork Fri Feb 28 03:38:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 11411467 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A265514E3 for ; Fri, 28 Feb 2020 03:38:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 61EA8246A0 for ; Fri, 28 Feb 2020 03:38:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 61EA8246A0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 826E66B0008; Thu, 27 Feb 2020 22:38:49 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7FD996B000A; Thu, 27 Feb 2020 22:38:49 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 73B6F6B000C; Thu, 27 Feb 2020 22:38:49 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0028.hostedemail.com [216.40.44.28]) by kanga.kvack.org (Postfix) with ESMTP id 5BC8C6B0008 for ; Thu, 27 Feb 2020 22:38:49 -0500 (EST) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 38E698248047 for ; Fri, 28 Feb 2020 03:38:49 +0000 (UTC) X-FDA: 76538129178.21.plot64_12b5cd1bbdd56 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,ying.huang@intel.com,,RULES_HIT:30003:30054:30064:30091,0,RBL:192.55.52.151:@intel.com:.lbl8.mailshell.net-64.95.201.95 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: plot64_12b5cd1bbdd56 X-Filterd-Recvd-Size: 6866 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Fri, 28 Feb 2020 03:38:48 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 27 Feb 2020 19:38:47 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,493,1574150400"; d="scan'208";a="232107402" Received: from yhuang-dev.sh.intel.com ([10.239.159.23]) by orsmga008.jf.intel.com with ESMTP; 27 Feb 2020 19:38:44 -0800 From: "Huang, Ying" To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , David Hildenbrand , Mel Gorman , Vlastimil Babka , Zi Yan , Michal Hocko , Peter Zijlstra , Dave Hansen , Minchan Kim , Johannes Weiner , Hugh Dickins Subject: [RFC 3/3] mm: Discard lazily freed pages when migrating Date: Fri, 28 Feb 2020 11:38:19 +0800 Message-Id: <20200228033819.3857058-4-ying.huang@intel.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200228033819.3857058-1-ying.huang@intel.com> References: <20200228033819.3857058-1-ying.huang@intel.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Huang Ying MADV_FREE is a lazy free mechanism in Linux. According to the manpage of mavise(2), the semantics of MADV_FREE is, The application no longer requires the pages in the range specified by addr and len. The kernel can thus free these pages, but the freeing could be delayed until memory pressure occurs. ... Originally, the pages freed lazily by MADV_FREE will only be freed really by page reclaiming when there is memory pressure or when unmapping the address range. In addition to that, there's another opportunity to free these pages really, when we try to migrate them. The main value to do that is to avoid to create the new memory pressure immediately if possible. Instead, even if the pages are required again, they will be allocated gradually on demand. That is, the memory will be allocated lazily when necessary. This follows the common philosophy in the Linux kernel, allocate resources lazily on demand. Signed-off-by: "Huang, Ying" Cc: David Hildenbrand Cc: Mel Gorman Cc: Vlastimil Babka Cc: Zi Yan Cc: Michal Hocko Cc: Peter Zijlstra Cc: Dave Hansen Cc: Minchan Kim Cc: Johannes Weiner Cc: Hugh Dickins --- include/linux/migrate.h | 4 ++++ mm/huge_memory.c | 20 +++++++++++++++----- mm/migrate.c | 16 +++++++++++++++- mm/rmap.c | 10 ++++++++++ 4 files changed, 44 insertions(+), 6 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 72120061b7d4..2c6cf985a8d3 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -14,8 +14,12 @@ typedef void free_page_t(struct page *page, unsigned long private); * Return values from addresss_space_operations.migratepage(): * - negative errno on page migration failure; * - zero on page migration success; + * + * __unmap_and_move() can also return 1 to indicate the page can be + * discarded instead of migrated. */ #define MIGRATEPAGE_SUCCESS 0 +#define MIGRATEPAGE_DISCARD 1 enum migrate_reason { MR_COMPACTION, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index b1e069e68189..b64f356ab77e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3063,11 +3063,21 @@ void set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, pmdval = pmdp_invalidate(vma, address, pvmw->pmd); if (pmd_dirty(pmdval)) set_page_dirty(page); - entry = make_migration_entry(page, pmd_write(pmdval)); - pmdswp = swp_entry_to_pmd(entry); - if (pmd_soft_dirty(pmdval)) - pmdswp = pmd_swp_mksoft_dirty(pmdswp); - set_pmd_at(mm, address, pvmw->pmd, pmdswp); + /* Clean lazyfree page, discard instead of migrate */ + if (PageLazyFree(page) && !PageDirty(page)) { + pmd_clear(pvmw->pmd); + zap_deposited_table(mm, pvmw->pmd); + /* Invalidate as we cleared the pmd */ + mmu_notifier_invalidate_range(mm, address, + address + HPAGE_PMD_SIZE); + add_mm_counter(mm, MM_ANONPAGES, -HPAGE_PMD_NR); + } else { + entry = make_migration_entry(page, pmd_write(pmdval)); + pmdswp = swp_entry_to_pmd(entry); + if (pmd_soft_dirty(pmdval)) + pmdswp = pmd_swp_mksoft_dirty(pmdswp); + set_pmd_at(mm, address, pvmw->pmd, pmdswp); + } page_remove_rmap(page, true); put_page(page); } diff --git a/mm/migrate.c b/mm/migrate.c index 981f8374a6ef..b7e7d18af94c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1122,6 +1122,11 @@ static int __unmap_and_move(struct page *page, struct page *newpage, goto out_unlock_both; } page_was_mapped = 1; + /* Clean lazyfree page, discard instead of migrate */ + if (PageLazyFree(page) && !PageDirty(page)) { + rc = MIGRATEPAGE_DISCARD; + goto out_unlock_both; + } } if (!page_mapped(page)) @@ -1242,7 +1247,16 @@ static ICE_noinline int unmap_and_move(new_page_t get_new_page, num_poisoned_pages_inc(); } } else { - if (rc != -EAGAIN) { + /* + * If page is discard instead of migrated, release + * reference grabbed during isolation, free the new + * page. For the caller, this is same as migrating + * successfully. + */ + if (rc == MIGRATEPAGE_DISCARD) { + put_page(page); + rc = MIGRATEPAGE_SUCCESS; + } else if (rc != -EAGAIN) { if (likely(!__PageMovable(page))) { putback_lru_page(page); goto put_new; diff --git a/mm/rmap.c b/mm/rmap.c index 1dcbb1771dd7..bb52883f7b2d 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1569,6 +1569,16 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, swp_entry_t entry; pte_t swp_pte; + /* Clean lazyfree page, discard instead of migrate */ + if (PageLazyFree(page) && !PageDirty(page) && + !(flags & TTU_SPLIT_FREEZE)) { + /* Invalidate as we cleared the pte */ + mmu_notifier_invalidate_range(mm, + address, address + PAGE_SIZE); + dec_mm_counter(mm, MM_ANONPAGES); + goto discard; + } + if (arch_unmap_one(mm, vma, address, pteval) < 0) { set_pte_at(mm, address, pvmw.pte, pteval); ret = false;