From patchwork Wed Aug 3 01:21:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12935113 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 522A1C19F2B for ; Wed, 3 Aug 2022 01:22:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5CF888E0005; Tue, 2 Aug 2022 21:22:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 57C2A8E0001; Tue, 2 Aug 2022 21:22:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A59C8E0005; Tue, 2 Aug 2022 21:22:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2DE428E0001 for ; Tue, 2 Aug 2022 21:22:08 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1040C1A0FC6 for ; Wed, 3 Aug 2022 01:22:08 +0000 (UTC) X-FDA: 79756530336.07.4166AA5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 8E7A5160110 for ; Wed, 3 Aug 2022 01:22:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1659489727; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qSNm0hC0oMVbN7mCZyyiRtCdb6eVWmsQyqPCRK6Htxk=; b=FVRg48ooc4bSz1ZlaYNmofmCNxvjMG0mPuy0Ua7p6Y+JGO2W3nUWgQmBvc78MOAnKxjyEB F+dCW/tOrIvIHnRX3Cv4T7i7YfzKH3N4CMMcXnSEjNTiOREdoz/GOVWFRSp05AmjXiFtZJ p7JPdMZlookn4wmnB1+qIOEbVpRy0JE= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-634-jkzMWabINhaor29FUPHV5Q-1; Tue, 02 Aug 2022 21:22:06 -0400 X-MC-Unique: jkzMWabINhaor29FUPHV5Q-1 Received: by mail-qt1-f197.google.com with SMTP id f1-20020ac84641000000b0031ecb35e4d1so9980964qto.2 for ; Tue, 02 Aug 2022 18:22:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qSNm0hC0oMVbN7mCZyyiRtCdb6eVWmsQyqPCRK6Htxk=; b=0DG2yfJPSYLMoom22FLw/dyiuqbJfiNkBadFO3PPbuNdfrcD3Kk88YqB2raC9gljRM o3a3inz1mwNTYPUEL9AkB2r6QdL+zeUI+YiKYneoc1y5D6QHwD75IQa2wZuz+ELkMHmL iG5MG56wYrKO3hTGc/Ym659vl6PSvWKht4TPdtQlqaRUcy1kesie7IjcoUOE2GDXm/aZ wk4NN1qkxpPx7TA/gKFAAE1An6+cgG1AjcCqpbQiBl590p/wmecHev2LYAVSbpHcan5m g2MI3mYhDZl7JusYeqyndRI6ZV8Zn2R1XORucPKSoyNzgNXD+VgNfV/iN6COaNuH74PQ MYXQ== X-Gm-Message-State: ACgBeo26D385qOWtlaij40iB6xxvyw3Qv0MRs3/88z96mgqmDqyOqgD4 brPX16eK8LLMz5EGCgL4tmQ+SR4Bq/zGmLeQ8SZofAISAAG6ZAEXE9Eq/afUCvxpDAhPKRlUePc VY3kbhe924xg= X-Received: by 2002:a05:6214:1c88:b0:472:a7e2:bef4 with SMTP id ib8-20020a0562141c8800b00472a7e2bef4mr20152395qvb.32.1659489725551; Tue, 02 Aug 2022 18:22:05 -0700 (PDT) X-Google-Smtp-Source: AA6agR4UH8F044fQ08lVU/hx51p2rYfphp5Z9tYfImkYnzCIYCnCwy8y/F7hg34O3s3p80m6z+rB/A== X-Received: by 2002:a05:6214:1c88:b0:472:a7e2:bef4 with SMTP id ib8-20020a0562141c8800b00472a7e2bef4mr20152382qvb.32.1659489725312; Tue, 02 Aug 2022 18:22:05 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id s16-20020a05622a1a9000b0031ecce4077fsm10188675qtc.31.2022.08.02.18.22.03 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 02 Aug 2022 18:22:04 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrea Arcangeli , Andi Kleen , Andrew Morton , David Hildenbrand , Hugh Dickins , Huang Ying , peterx@redhat.com, Nadav Amit , "Kirill A . Shutemov" , Vlastimil Babka Subject: [PATCH 2/2] mm: Remember young bit for page migrations Date: Tue, 2 Aug 2022 21:21:59 -0400 Message-Id: <20220803012159.36551-3-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220803012159.36551-1-peterx@redhat.com> References: <20220803012159.36551-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FVRg48oo; spf=pass (imf08.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659489727; a=rsa-sha256; cv=none; b=A8i097KWDGfII32ztY5z7zlKwARzoF/5jF/cGAsRunWPyk2dQldxzL8HZPCY2Xta5PUFWT Slp+ryF5OBXxOrzvmaJCCXGhO8mTUWWkixw9+HY3oBNmfMoFuzl8StY0Hea5Nj/cifIFWq ahJnAE1gn6FvJWHHSRb+9gAZrI+ypvg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659489727; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qSNm0hC0oMVbN7mCZyyiRtCdb6eVWmsQyqPCRK6Htxk=; b=xkmscMMlg3b9kaVifzAmRnfRFoubTNOHbp8SnbvhFEUHa4F2worXArcmIvS8heNbER80Sy +C4sMWErUj4rfNGlDlWpfRvXPdGbEjspc2a659GVfAVcrm+f2GHVk7cd5bwlPknjny5u8+ XnojqFjCHBVFyOwfkBU7zr8o/nWZ9yY= X-Stat-Signature: jsay38hzy9eewjkkxki9ougmrzkszczx X-Rspamd-Queue-Id: 8E7A5160110 Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FVRg48oo; spf=pass (imf08.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1659489727-33738 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When page migration happens, we always ignore the young bit settings in the old pgtable, and marking the page as old in the new page table using either pte_mkold() or pmd_mkold(). That's fine from functional-wise, but that's not friendly to page reclaim because the moving page can be actively accessed within the procedure. Not to mention hardware setting the young bit can bring quite some overhead on some systems, e.g. x86_64 needs a few hundreds nanoseconds to set the bit. Actually we can easily remember the young bit configuration and recover the information after the page is migrated. To achieve it, define a new bit in the migration swap offset field to show whether the old pte has young bit set or not. Then when removing/recovering the migration entry, we can recover the young bit even if the page changed. One thing to mention is that here we used max_swapfile_size() to detect how many swp offset bits we have, and we'll only enable this feature if we know the swp offset can be big enough to store both the PFN value and the young bit. Otherwise the young bit is dropped like before. Signed-off-by: Peter Xu Reported-by: kernel test robot Reported-by: kernel test robot --- include/linux/swapops.h | 49 +++++++++++++++++++++++++++++++++++++++++ mm/huge_memory.c | 10 +++++++-- mm/migrate.c | 4 +++- mm/migrate_device.c | 2 ++ mm/rmap.c | 3 ++- 5 files changed, 64 insertions(+), 4 deletions(-) diff --git a/include/linux/swapops.h b/include/linux/swapops.h index 1d17e4bb3d2f..9ddede3790a4 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -8,6 +8,8 @@ #ifdef CONFIG_MMU +#include + /* * swapcache pages are stored in the swapper_space radix tree. We want to * get good packing density in that tree, so the index should be dense in @@ -35,6 +37,16 @@ #endif #define SWP_PFN_MASK ((1UL << SWP_PFN_BITS) - 1) +/** + * Migration swap entry specific bitfield definitions. + * + * @SWP_MIG_YOUNG_BIT: Whether the page used to have young bit set + * + * Note: these bits will be used only if there're free bits in arch + * specific swp offset field. + */ +#define SWP_MIG_YOUNG_BIT (1UL << SWP_PFN_BITS) + static inline bool is_pfn_swap_entry(swp_entry_t entry); /* Clear all flags but only keep swp_entry_t related information */ @@ -265,6 +277,33 @@ static inline swp_entry_t make_writable_migration_entry(pgoff_t offset) return swp_entry(SWP_MIGRATION_WRITE, offset); } +static inline bool migration_entry_supports_young(void) +{ + /* + * max_swapfile_size() returns the max supported swp-offset plus 1. + * We can support the migration young bit only if the pfn swap + * entry has the offset larger than storing the PFN value, then it + * means there's extra bit(s) where we can store the young bit. + */ + return max_swapfile_size() > SWP_MIG_YOUNG_BIT; +} + +static inline swp_entry_t make_migration_entry_young(swp_entry_t entry) +{ + if (migration_entry_supports_young()) + return swp_entry(swp_type(entry), + swp_offset(entry) | SWP_MIG_YOUNG_BIT); + return entry; +} + +static inline bool is_migration_entry_young(swp_entry_t entry) +{ + if (migration_entry_supports_young()) + return swp_offset(entry) & SWP_MIG_YOUNG_BIT; + /* Keep the old behavior of aging page after migration */ + return false; +} + extern void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, spinlock_t *ptl); extern void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, @@ -311,6 +350,16 @@ static inline int is_readable_migration_entry(swp_entry_t entry) return 0; } +static inline swp_entry_t make_migration_entry_young(swp_entry_t entry) +{ + return entry; +} + +static inline bool is_migration_entry_young(swp_entry_t entry) +{ + return false; +} + #endif typedef unsigned long pte_marker; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 29e3628687a6..131fe5754d8f 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2088,7 +2088,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, write = is_writable_migration_entry(entry); if (PageAnon(page)) anon_exclusive = is_readable_exclusive_migration_entry(entry); - young = false; + young = is_migration_entry_young(entry); soft_dirty = pmd_swp_soft_dirty(old_pmd); uffd_wp = pmd_swp_uffd_wp(old_pmd); } else { @@ -2146,6 +2146,8 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, else swp_entry = make_readable_migration_entry( page_to_pfn(page + i)); + if (young) + swp_entry = make_migration_entry_young(swp_entry); entry = swp_entry_to_pte(swp_entry); if (soft_dirty) entry = pte_swp_mksoft_dirty(entry); @@ -3148,6 +3150,8 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, entry = make_readable_exclusive_migration_entry(page_to_pfn(page)); else entry = make_readable_migration_entry(page_to_pfn(page)); + if (pmd_young(pmdval)) + entry = make_migration_entry_young(entry); pmdswp = swp_entry_to_pmd(entry); if (pmd_soft_dirty(pmdval)) pmdswp = pmd_swp_mksoft_dirty(pmdswp); @@ -3173,13 +3177,15 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) entry = pmd_to_swp_entry(*pvmw->pmd); get_page(new); - pmde = pmd_mkold(mk_huge_pmd(new, READ_ONCE(vma->vm_page_prot))); + pmde = mk_huge_pmd(new, READ_ONCE(vma->vm_page_prot)); if (pmd_swp_soft_dirty(*pvmw->pmd)) pmde = pmd_mksoft_dirty(pmde); if (is_writable_migration_entry(entry)) pmde = maybe_pmd_mkwrite(pmde, vma); if (pmd_swp_uffd_wp(*pvmw->pmd)) pmde = pmd_wrprotect(pmd_mkuffd_wp(pmde)); + if (!is_migration_entry_young(entry)) + pmde = pmd_mkold(pmde); if (PageAnon(new)) { rmap_t rmap_flags = RMAP_COMPOUND; diff --git a/mm/migrate.c b/mm/migrate.c index 1649270bc1a7..62cb3a9451de 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -199,7 +199,7 @@ static bool remove_migration_pte(struct folio *folio, #endif folio_get(folio); - pte = pte_mkold(mk_pte(new, READ_ONCE(vma->vm_page_prot))); + pte = mk_pte(new, READ_ONCE(vma->vm_page_prot)); if (pte_swp_soft_dirty(*pvmw.pte)) pte = pte_mksoft_dirty(pte); @@ -207,6 +207,8 @@ static bool remove_migration_pte(struct folio *folio, * Recheck VMA as permissions can change since migration started */ entry = pte_to_swp_entry(*pvmw.pte); + if (!is_migration_entry_young(entry)) + pte = pte_mkold(pte); if (is_writable_migration_entry(entry)) pte = maybe_mkwrite(pte, vma); else if (pte_swp_uffd_wp(*pvmw.pte)) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 7feeb447e3b9..fd8daf45c1a6 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -221,6 +221,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, else entry = make_readable_migration_entry( page_to_pfn(page)); + if (pte_young(pte)) + entry = make_migration_entry_young(entry); swp_pte = swp_entry_to_pte(entry); if (pte_present(pte)) { if (pte_soft_dirty(pte)) diff --git a/mm/rmap.c b/mm/rmap.c index af775855e58f..605fb37ae95e 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2065,7 +2065,8 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, else entry = make_readable_migration_entry( page_to_pfn(subpage)); - + if (pte_young(pteval)) + entry = make_migration_entry_young(entry); swp_pte = swp_entry_to_pte(entry); if (pte_soft_dirty(pteval)) swp_pte = pte_swp_mksoft_dirty(swp_pte);