From patchwork Mon Mar 11 15:00:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13588938 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7CFFC5475B for ; Mon, 11 Mar 2024 15:01:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CE17E6B00AF; Mon, 11 Mar 2024 11:01:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C8F546B00B0; Mon, 11 Mar 2024 11:01:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B2E8D6B00B1; Mon, 11 Mar 2024 11:01:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A09E36B00AF for ; Mon, 11 Mar 2024 11:01:29 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6BBA280C64 for ; Mon, 11 Mar 2024 15:01:29 +0000 (UTC) X-FDA: 81885071898.26.5A052B4 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf21.hostedemail.com (Postfix) with ESMTP id 00A211C0022 for ; Mon, 11 Mar 2024 15:01:24 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf21.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710169285; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UI5PRAj7U+kLDNSybnSkxpbypRwhNSJXjiMivm+ZKt8=; b=IC9gTmPTh2MMYim35t3xajhpqA2Oa4/d9DsZsFBWCuRCp4sNO6eTXYFMumSIAHoxoOzmG5 sChe2MeEkRnIDc9w8Qd7GXIDm8nGSSAm4dEet+wGOeks4EvW3krfuxbCE+f3YOHiQPJvBW ABRWYLgT0YhBQ8dI5rV5rb+f7akoHJc= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf21.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710169285; a=rsa-sha256; cv=none; b=E6OieJKgMEVQPSyWxOFfq/Zb2gEb84LBtO+hZ92p6ojpdtLTi360v6n63pyBXPcj5x4IQt bmySwvuDC8BcHZS0lEta4HYjUbcN3A2vhWsc0sPyPMqSD8SreE6ktLh6+7X8jpsSAXhGyP e7iZ08YnA4r/QEYgaB/dQVEEoiSDn/Q= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0FEE3153B; Mon, 11 Mar 2024 08:02:01 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 267053F64C; Mon, 11 Mar 2024 08:01:22 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , David Hildenbrand , Matthew Wilcox , Huang Ying , Gao Xiang , Yu Zhao , Yang Shi , Michal Hocko , Kefeng Wang , Barry Song <21cnbao@gmail.com>, Chris Li Cc: Ryan Roberts , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD Date: Mon, 11 Mar 2024 15:00:58 +0000 Message-Id: <20240311150058.1122862-7-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240311150058.1122862-1-ryan.roberts@arm.com> References: <20240311150058.1122862-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 00A211C0022 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 899nx7o7ydgdo9kyk57aoy1ait1ffujt X-HE-Tag: 1710169284-119422 X-HE-Meta: U2FsdGVkX1/UkSvhLrgJvg9BxLjOyd1NP68LcIqLl3qW9thHKkQENeEgHbB9BhfTg8Zdrcc3ooZS8/jlKn96kzMZ0gXcA3tojxNxl2FxIwq7kzKtvUnpKn9g1hBeRyLVRNNfUwjBO+VrufgL+FUsgxfWBNW8RCtCB7LNUH+GWmMTDdopsXlITebfrrjp0L9HZekoC+c2yq57fOLagyE3apnzRikAtbOhAX/t7FuobX2ayuR8gsV7NeXMUCDAu0AXvieQx0UJ6h694McUQDQKpAPKihVv3CwDJnWpicbIoJOIWz3ZUyeRdOc+7y0RF/iMSGCVS07UNgGgF/TXuq5iuBpw5lmQ2fV+zRV5znGcvvpTRq5mU11To8gL6pLXHU1brHJjmzcehGMt6q4QT+gSE2N1ULqthLjy8rvni5+Vv8z/sO3IRlysSRuCncT/Dg0L8MwGgfwdKNyoWhMX++hK2yhx1piGFpWkY3BhHpnaBeSmDbKsIvsZ56eUmuyW+qXesq5+f65qgdAO5P7EuBjx7SmPx8tV2wfCrejSJVmjv+s5Y6lyjTXx4SrEMeWPGv9rOgQlHMLf5DuY+W5hB+O4xBiqnvaA26kgC2yL1rhdfp0pTK+0/EKB8Ng+4F56SRdam2eP0j7ASg32un2bF6mECMHk1eLhn0pVwQene0FNVKZXRtlkc/tTWxWD896MfVxN9TH5qKwOQWwIrdosnlyUEFhumJ5enedOo8/+XkLCoKZpNZcMIflrO+WFK5qDZQW+i8El32UZg6AFJy0fFgelVtDyp/0PvFRKYHfMN8lz1OiiQwRN0O5XKUl3JhTVeOvqbHmjkjnFp35UUiBLZ0k3iINTu+Qrztq59T1FxUYDtuctWmV7jgEDLilBlADQqnQ9zyz68ZWuBFJaeFMVNl4uCQe5Cw92fosLQ5mmIRD7HPYjloFB5DnvuiTDV/bxS+++3uRGOlcr0LIjvS4H3Jz PuLL+419 U8bm6RTo0D5GU3EwdtVFqaZra42QNZNU0lsxSlilM9oTJ2GLcykWdNVzmlOE1TrKwhx2n+UFEvpKBDtgwr8OnJwiPtYMdFFz9Q55vogwx5SdFyx7SODtVEHz4LcPmHJgh149gi8wHzSeAjKF5qgzs/8fD0w6tmrLNMX3ibZwaGxfUwY4Y+ZlXi62nU8ojHcrFpqWXIC+g7GYJEic= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Rework madvise_cold_or_pageout_pte_range() to avoid splitting any large folio that is fully and contiguously mapped in the pageout/cold vm range. This change means that large folios will be maintained all the way to swap storage. This both improves performance during swap-out, by eliding the cost of splitting the folio, and sets us up nicely for maintaining the large folio when it is swapped back in (to be covered in a separate series). Folios that are not fully mapped in the target range are still split, but note that behavior is changed so that if the split fails for any reason (folio locked, shared, etc) we now leave it as is and move to the next pte in the range and continue work on the proceeding folios. Previously any failure of this sort would cause the entire operation to give up and no folios mapped at higher addresses were paged out or made cold. Given large folios are becoming more common, this old behavior would have likely lead to wasted opportunities. While we are at it, change the code that clears young from the ptes to use ptep_test_and_clear_young(), which is more efficent than get_and_clear/modify/set, especially for contpte mappings on arm64, where the old approach would require unfolding/refolding and the new approach can be done in place. Signed-off-by: Ryan Roberts Reviewed-by: Barry Song --- mm/madvise.c | 89 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 38 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 547dcd1f7a39..56c7ba7bd558 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -336,6 +336,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, LIST_HEAD(folio_list); bool pageout_anon_only_filter; unsigned int batch_count = 0; + int nr; if (fatal_signal_pending(current)) return -EINTR; @@ -423,7 +424,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, return 0; flush_tlb_batched_pending(mm); arch_enter_lazy_mmu_mode(); - for (; addr < end; pte++, addr += PAGE_SIZE) { + for (; addr < end; pte += nr, addr += nr * PAGE_SIZE) { + nr = 1; ptent = ptep_get(pte); if (++batch_count == SWAP_CLUSTER_MAX) { @@ -447,55 +449,66 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, continue; /* - * Creating a THP page is expensive so split it only if we - * are sure it's worth. Split it if we are only owner. + * If we encounter a large folio, only split it if it is not + * fully mapped within the range we are operating on. Otherwise + * leave it as is so that it can be swapped out whole. If we + * fail to split a folio, leave it in place and advance to the + * next pte in the range. */ if (folio_test_large(folio)) { - int err; - - if (folio_estimated_sharers(folio) > 1) - break; - if (pageout_anon_only_filter && !folio_test_anon(folio)) - break; - if (!folio_trylock(folio)) - break; - folio_get(folio); - arch_leave_lazy_mmu_mode(); - pte_unmap_unlock(start_pte, ptl); - start_pte = NULL; - err = split_folio(folio); - folio_unlock(folio); - folio_put(folio); - if (err) - break; - start_pte = pte = - pte_offset_map_lock(mm, pmd, addr, &ptl); - if (!start_pte) - break; - arch_enter_lazy_mmu_mode(); - pte--; - addr -= PAGE_SIZE; - continue; + const fpb_t fpb_flags = FPB_IGNORE_DIRTY | + FPB_IGNORE_SOFT_DIRTY; + int max_nr = (end - addr) / PAGE_SIZE; + + nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, + fpb_flags, NULL); + + if (nr < folio_nr_pages(folio)) { + int err; + + if (folio_estimated_sharers(folio) > 1) + continue; + if (pageout_anon_only_filter && !folio_test_anon(folio)) + continue; + if (!folio_trylock(folio)) + continue; + folio_get(folio); + arch_leave_lazy_mmu_mode(); + pte_unmap_unlock(start_pte, ptl); + start_pte = NULL; + err = split_folio(folio); + folio_unlock(folio); + folio_put(folio); + if (err) + continue; + start_pte = pte = + pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!start_pte) + break; + arch_enter_lazy_mmu_mode(); + nr = 0; + continue; + } } /* * Do not interfere with other mappings of this folio and - * non-LRU folio. + * non-LRU folio. If we have a large folio at this point, we + * know it is fully mapped so if its mapcount is the same as its + * number of pages, it must be exclusive. */ - if (!folio_test_lru(folio) || folio_mapcount(folio) != 1) + if (!folio_test_lru(folio) || + folio_mapcount(folio) != folio_nr_pages(folio)) continue; if (pageout_anon_only_filter && !folio_test_anon(folio)) continue; - VM_BUG_ON_FOLIO(folio_test_large(folio), folio); - - if (!pageout && pte_young(ptent)) { - ptent = ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm); - ptent = pte_mkold(ptent); - set_pte_at(mm, addr, pte, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); + if (!pageout) { + for (; nr != 0; nr--, pte++, addr += PAGE_SIZE) { + if (ptep_test_and_clear_young(vma, addr, pte)) + tlb_remove_tlb_entry(tlb, pte, addr); + } } /*