From patchwork Mon Mar 6 09:22:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13160759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB919C61DA4 for ; Mon, 6 Mar 2023 09:22:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 01DC06B0074; Mon, 6 Mar 2023 04:22:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F108C6B007B; Mon, 6 Mar 2023 04:22:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DDA0B6B007D; Mon, 6 Mar 2023 04:22:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CEF886B0074 for ; Mon, 6 Mar 2023 04:22:25 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 93B6D40222 for ; Mon, 6 Mar 2023 09:22:25 +0000 (UTC) X-FDA: 80537932650.10.A7D941F Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf22.hostedemail.com (Postfix) with ESMTP id 95CDBC0012 for ; Mon, 6 Mar 2023 09:22:23 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Q0jvkHYd; spf=pass (imf22.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678094543; a=rsa-sha256; cv=none; b=a4ulrFb8GmVDuCoB29oduZyAC+sQFpDx5pNhyxv+DjJnKguSBg0l5jwhWv/4Y3qN9yfZwq +38szfvlPylkBeqHlRGNb5UiM7ClAmyaomcT1VogWo8pUehYbCASt8nBwsA2rQsbNxGbfe 7BurSUp2tI9FS/8xR4yhaFK2/wDP98Y= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Q0jvkHYd; spf=pass (imf22.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678094543; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rfTHxPoxGggk3s18axmB19Urf5GRweKPAMJqctjpi3g=; b=8XZvZJz1LKIw4HmRhuN+hbQ2dzEbJAxzPZZtE20TEy9/I0JCnwBhJSbK6CQip8UeQQZnxd 1S7TD0gv9oP4CfUP99VQfG9QlsowQuJz2D/lcXXj35QXd1kFGAqXaceWMJivjFH1KyDcYQ /gjrJl9Q8Yp4SeWsFuiDQiJds+NZ+TM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678094543; x=1709630543; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=71/4/wxaPV0XPVnrmXXtIkG0KxHDkOgrfZzOHhO22ss=; b=Q0jvkHYdS80NQi5s6ekiG8AqZiDWRPxLGkNma925pNvYX5gcVyBWqHA9 WTCnRLt+LATGxhrCL/1gS9361gmpkNcLmY2PGLr1tBiSiwDandrEjBnf+ VEYTpau6Wir53JnlOGQomW/ivx9Mj/tp4vt3nyR0eE3l4UNTTOlAvoM5v ZuAeYjQUj4s/Gw2d79z8tVFmOFrIaFLxExsZntC/B8F2U4IH8WlQ5lkE9 MJnsz1NpoWSwE5ArjO28PLIcAD1F0KUI2JhcBRgmk/fbVl7oB1gJAERq1 HG8FZGEzgjAqskRHWswgcrq2SilIHRlxBxIQiwbkpmRbEQs3f5mtSXb75 g==; X-IronPort-AV: E=McAfee;i="6500,9779,10640"; a="363124026" X-IronPort-AV: E=Sophos;i="5.98,236,1673942400"; d="scan'208";a="363124026" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 01:22:21 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10640"; a="799932460" X-IronPort-AV: E=Sophos;i="5.98,236,1673942400"; d="scan'208";a="799932460" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by orsmga004.jf.intel.com with ESMTP; 06 Mar 2023 01:22:19 -0800 From: Yin Fengwei To: linux-mm@kvack.org, akpm@linux-foundation.org, willy@infradead.org, mike.kravetz@oracle.com, sidhartha.kumar@oracle.com, naoya.horiguchi@nec.com, jane.chu@oracle.com, david@redhat.com Cc: fengwei.yin@intel.com Subject: [PATCH v3 3/5] rmap: cleanup exit path of try_to_unmap_one_page() Date: Mon, 6 Mar 2023 17:22:57 +0800 Message-Id: <20230306092259.3507807-4-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230306092259.3507807-1-fengwei.yin@intel.com> References: <20230306092259.3507807-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 95CDBC0012 X-Rspamd-Server: rspam01 X-Stat-Signature: t789iybqbk8a75pmxx8xko8arautt89f X-HE-Tag: 1678094543-991615 X-HE-Meta: U2FsdGVkX198NY2I6BiRm+jQv5jfN5I9bBdC1O6YXGzG3yPgFSTUXx8q9frou+XPqMTmKrr9hfyIH7upOvNhcB+V8ioeHQhdOWPd+GCM4LWtO1RP7K2JzaahSM2E5bzE9Bu1snXxgQi7dfRH0JYSIH477WjxS66LPxi5Q769+c9BkH248J+fqne2jr2zepLnF9osfkWe69iGc8uJ+8jqrWUuYLSuJeWTPNoDes6DArla6Qm1FrxodN/YOXeR9aAMDS7vXoM9KkhYQpbLKtybXe7g8y0QTaS07xuriSEDe2EeSQ18oY2JgQFdmqQLFU+TEoSPq+WJEDseIbB2N1YB9cBHG5rL45dYi928WCA3leYU0QU7brSLitvW5ttOVaK6SmkWLoC1i91NcjUWJOulhwSskQIJ4RQ3sphhImqlvKrtoABh4yPxZiJhtD9EuFd/ZTNbXKYr1aVkSz4Emp/Oj9ogVQ4WU3rlhJCvu6PHkJDp/aJqIuRg5J9uBuIWxjaH2yJ0kUGbCaIav8X6fXOQV6vzyKMXFvHf0gthLuFmg9axcfx0SaeZ29ZOrSEWSMAoNYzivbRk721sEVLg5RciaTLU2QE8ibmPvEuIdYZE9hYE5ZAMLjj/SMsZyKMwvWsTrtIYOyE0iDer/fdTShLBF7jWJ2W/TE/JROM2xIaks2TiLMft9zBeVkpQY82SQDBXIWz3m9E7KWYFONEQ/gBnkjKEVZOuanJ8ujHh/pfb5myoow8BFjnJZmG1fsdXMeZyJYuGa/Aq4+dIfwZ0MS9Z7fHmV+/MMbvcWSKd+vH2pAkoyH7yJ+Q+IJ+pJ3La9pgbtcowGqXPpgwTu+dCLDfd0jU6tddXzcR5yPULz3fd2OOazLapQbffleAZGuA/6MmoWW4Urm6HcFYmxmF9jsOcmzecToZGhC4fg37bHRS65dRkPdsbPIPxUham7Q9JWV7GH3EElwpDVG2g2+Li6Wa LhlFmq3v KHwj16/nS0XCK4qEMQTwN9/H7470bR/NsaUIRYDFttfjUfcMoesTOH+jWDJEYY9U2y/7zFGdesPzz//CKiE413z/aHiVSS+dWH4AD421dLkQiPCU8JgLCblYFB82mg2Skja65bOjUU+KabHkzQdjPCnWKGAuFC9xk1Lis0x0+OMxnwvJKPhgChK5bOI8OdnYlL36fWNe6PytE41nGwF79WbFx8w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Cleanup exit path of try_to_unmap_one_page() by removing some duplicated code. Move page_vma_mapped_walk_done() back to try_to_unmap_one(). Change subpage to page as folio has no concept of subpage. Signed-off-by: Yin Fengwei --- mm/rmap.c | 74 ++++++++++++++++++++++--------------------------------- 1 file changed, 30 insertions(+), 44 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 013643122d0c..836cfc13cf9d 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1528,7 +1528,7 @@ static bool try_to_unmap_one_hugetlb(struct folio *folio, * * See Documentation/mm/mmu_notifier.rst */ - page_remove_rmap(&folio->page, vma, folio_test_hugetlb(folio)); + page_remove_rmap(&folio->page, vma, true); /* No VM_LOCKED set in vma->vm_flags for hugetlb. So not * necessary to call mlock_drain_local(). */ @@ -1543,15 +1543,13 @@ static bool try_to_unmap_one_page(struct folio *folio, struct page_vma_mapped_walk pvmw, unsigned long address, enum ttu_flags flags) { - bool anon_exclusive, ret = true; - struct page *subpage; + bool anon_exclusive; + struct page *page; struct mm_struct *mm = vma->vm_mm; pte_t pteval; - subpage = folio_page(folio, - pte_pfn(*pvmw.pte) - folio_pfn(folio)); - anon_exclusive = folio_test_anon(folio) && - PageAnonExclusive(subpage); + page = folio_page(folio, pte_pfn(*pvmw.pte) - folio_pfn(folio)); + anon_exclusive = folio_test_anon(folio) && PageAnonExclusive(page); flush_cache_page(vma, address, pte_pfn(*pvmw.pte)); /* Nuke the page table entry. */ @@ -1579,15 +1577,14 @@ static bool try_to_unmap_one_page(struct folio *folio, pte_install_uffd_wp_if_needed(vma, address, pvmw.pte, pteval); /* Set the dirty flag on the folio now the pte is gone. */ - if (pte_dirty(pteval)) + if (pte_dirty(pteval) && !folio_test_dirty(folio)) folio_mark_dirty(folio); /* Update high watermark before we lower rss */ update_hiwater_rss(mm); - if (PageHWPoison(subpage) && !(flags & TTU_HWPOISON)) { - pteval = swp_entry_to_pte(make_hwpoison_entry(subpage)); - dec_mm_counter(mm, mm_counter(&folio->page)); + if (PageHWPoison(page) && !(flags & TTU_HWPOISON)) { + pteval = swp_entry_to_pte(make_hwpoison_entry(page)); set_pte_at(mm, address, pvmw.pte, pteval); } else if (pte_unused(pteval) && !userfaultfd_armed(vma)) { /* @@ -1600,12 +1597,11 @@ static bool try_to_unmap_one_page(struct folio *folio, * migration) will not expect userfaults on already * copied pages. */ - dec_mm_counter(mm, mm_counter(&folio->page)); /* We have to invalidate as we cleared the pte */ mmu_notifier_invalidate_range(mm, address, address + PAGE_SIZE); } else if (folio_test_anon(folio)) { - swp_entry_t entry = { .val = page_private(subpage) }; + swp_entry_t entry = { .val = page_private(page) }; pte_t swp_pte; /* * Store the swap location in the pte. @@ -1614,12 +1610,10 @@ static bool try_to_unmap_one_page(struct folio *folio, if (unlikely(folio_test_swapbacked(folio) != folio_test_swapcache(folio))) { WARN_ON_ONCE(1); - ret = false; /* We have to invalidate as we cleared the pte */ mmu_notifier_invalidate_range(mm, address, address + PAGE_SIZE); - page_vma_mapped_walk_done(&pvmw); - goto discard; + goto exit; } /* MADV_FREE page check */ @@ -1651,7 +1645,6 @@ static bool try_to_unmap_one_page(struct folio *folio, /* Invalidate as we cleared the pte */ mmu_notifier_invalidate_range(mm, address, address + PAGE_SIZE); - dec_mm_counter(mm, MM_ANONPAGES); goto discard; } @@ -1659,43 +1652,30 @@ static bool try_to_unmap_one_page(struct folio *folio, * If the folio was redirtied, it cannot be * discarded. Remap the page to page table. */ - set_pte_at(mm, address, pvmw.pte, pteval); folio_set_swapbacked(folio); - ret = false; - page_vma_mapped_walk_done(&pvmw); - goto discard; + goto exit_restore_pte; } - if (swap_duplicate(entry) < 0) { - set_pte_at(mm, address, pvmw.pte, pteval); - ret = false; - page_vma_mapped_walk_done(&pvmw); - goto discard; - } + if (swap_duplicate(entry) < 0) + goto exit_restore_pte; + if (arch_unmap_one(mm, vma, address, pteval) < 0) { swap_free(entry); - set_pte_at(mm, address, pvmw.pte, pteval); - ret = false; - page_vma_mapped_walk_done(&pvmw); - goto discard; + goto exit_restore_pte; } /* See page_try_share_anon_rmap(): clear PTE first. */ - if (anon_exclusive && - page_try_share_anon_rmap(subpage)) { + if (anon_exclusive && page_try_share_anon_rmap(page)) { swap_free(entry); - set_pte_at(mm, address, pvmw.pte, pteval); - ret = false; - page_vma_mapped_walk_done(&pvmw); - goto discard; + goto exit_restore_pte; } + if (list_empty(&mm->mmlist)) { spin_lock(&mmlist_lock); if (list_empty(&mm->mmlist)) list_add(&mm->mmlist, &init_mm.mmlist); spin_unlock(&mmlist_lock); } - dec_mm_counter(mm, MM_ANONPAGES); inc_mm_counter(mm, MM_SWAPENTS); swp_pte = swp_entry_to_pte(entry); if (anon_exclusive) @@ -1706,8 +1686,7 @@ static bool try_to_unmap_one_page(struct folio *folio, swp_pte = pte_swp_mkuffd_wp(swp_pte); set_pte_at(mm, address, pvmw.pte, swp_pte); /* Invalidate as we cleared the pte */ - mmu_notifier_invalidate_range(mm, address, - address + PAGE_SIZE); + mmu_notifier_invalidate_range(mm, address, address + PAGE_SIZE); } else { /* * This is a locked file-backed folio, @@ -1720,11 +1699,16 @@ static bool try_to_unmap_one_page(struct folio *folio, * * See Documentation/mm/mmu_notifier.rst */ - dec_mm_counter(mm, mm_counter_file(&folio->page)); } discard: - return ret; + dec_mm_counter(vma->vm_mm, mm_counter(&folio->page)); + return true; + +exit_restore_pte: + set_pte_at(mm, address, pvmw.pte, pteval); +exit: + return false; } /* @@ -1802,8 +1786,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, pte_pfn(*pvmw.pte) - folio_pfn(folio)); ret = try_to_unmap_one_page(folio, vma, range, pvmw, address, flags); - if (!ret) + if (!ret) { + page_vma_mapped_walk_done(&pvmw); break; + } /* * No need to call mmu_notifier_invalidate_range() it has be @@ -1812,7 +1798,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, * * See Documentation/mm/mmu_notifier.rst */ - page_remove_rmap(subpage, vma, folio_test_hugetlb(folio)); + page_remove_rmap(subpage, vma, false); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio);