From patchwork Thu Aug 4 20:39:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12936685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F23C5C25B08 for ; Thu, 4 Aug 2022 20:40:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B45DD8E0005; Thu, 4 Aug 2022 16:40:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AD38E8E0001; Thu, 4 Aug 2022 16:40:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 838588E0005; Thu, 4 Aug 2022 16:40:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 727068E0001 for ; Thu, 4 Aug 2022 16:40:01 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 50A15A04F7 for ; Thu, 4 Aug 2022 20:40:01 +0000 (UTC) X-FDA: 79763077002.06.B32C74D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id D25EAC0129 for ; Thu, 4 Aug 2022 20:40:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1659645600; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yjbqitvPqSyZJdTsNvN60J+IesFSNFAJ7gz7O8lHdlw=; b=MYHn5HiNV5SlN0r0brZiuUkhWB8yUzzyDMCT11mH0kW3XpciIOehJ6hNwzjNWe4Y5ID5We OCwucKIVl+l+EbHImk0TyKOaG3es0sKsOaNncMjpEXvK8uYckNEq09OpxQC46qdmTpGu3e GDndeod5wpTJuG8noe9AUZ4WTCLTAN0= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-616-LDhC7UtdNy-CeyfYQnSV7Q-1; Thu, 04 Aug 2022 16:39:59 -0400 X-MC-Unique: LDhC7UtdNy-CeyfYQnSV7Q-1 Received: by mail-qk1-f200.google.com with SMTP id y17-20020a05620a25d100b006b66293d75aso436060qko.17 for ; Thu, 04 Aug 2022 13:39:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=yjbqitvPqSyZJdTsNvN60J+IesFSNFAJ7gz7O8lHdlw=; b=KWEbbJ9c+jOzvJb+OR/9ADVNbhW06sUq6YCR6PNnznvEomTAlNMrrPKbfOnvIkUgyO Vm6+VTOCH7HzAFkVpMlEaGEClpsYBTn95FRcC8OT0sbA72No/9GHlvcEHuyzcXMGg7/o WtuVoKE5tQ8beiD0JPDCCsQIMdAY/Q9jxVAr029EI+a0BwNZxLNC/J19uSyZm128x2zj 81Vc+vCUnTpIfpJmrRXpD5PqPTauRCq4F1lFh5cnYv3Hie4CH8XbGtPcI1+gUbMpy8R2 iGqQt+iodPaNPl41p9r+hNVY9AQxu9WmhAhxvzUOXN8qO1/YMG6GJTlD26M+xbZDmaUq CMTw== X-Gm-Message-State: ACgBeo33BsO42QwySc7Yv2lM3NxNsejhdTfr1IkD1OxTEJIPqZq4VA0x sN+TT1cvKAGIA/xIeugEqt1uCqe+jnziiDU6YYY5W3bwNQtyqvUlEXJUHB71Aoh2Y5zRUnfGhjc yD8m9ftgjbtTXKAlwFZCltjC9I5MFXGC8cU+WRRQIX8uc2E0xRHEBZQgBfaDx X-Received: by 2002:a05:622a:10a:b0:341:bba0:3b05 with SMTP id u10-20020a05622a010a00b00341bba03b05mr3101674qtw.648.1659645598597; Thu, 04 Aug 2022 13:39:58 -0700 (PDT) X-Google-Smtp-Source: AA6agR78YxyW7sWXeHw1v5VNrRrWnFysEzZiSyOl3pZEfHSEAaxw5+EQubb7GVQgqtDJNOthk7giRQ== X-Received: by 2002:a05:622a:10a:b0:341:bba0:3b05 with SMTP id u10-20020a05622a010a00b00341bba03b05mr3101643qtw.648.1659645598254; Thu, 04 Aug 2022 13:39:58 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id k19-20020ae9f113000000b006b5e50057basm1395266qkg.95.2022.08.04.13.39.56 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 04 Aug 2022 13:39:57 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Huang Ying , Andrea Arcangeli , David Hildenbrand , Minchan Kim , Andrew Morton , Vlastimil Babka , Nadav Amit , Hugh Dickins , Andi Kleen , peterx@redhat.com, "Kirill A . Shutemov" Subject: [PATCH v2 2/2] mm: Remember young/dirty bit for page migrations Date: Thu, 4 Aug 2022 16:39:52 -0400 Message-Id: <20220804203952.53665-3-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220804203952.53665-1-peterx@redhat.com> References: <20220804203952.53665-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659645601; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yjbqitvPqSyZJdTsNvN60J+IesFSNFAJ7gz7O8lHdlw=; b=wmzvUGfWnl/M7AUBIV6faP84tYvG4LBOIJc2N4QrpyiFwICVwbhaZIWfkpG0Arv030Kbf/ zmwRzZ7ypIQ/StsgrfCja6k+PbmxigMH5MTXenHLW4zqfGVcuBDUHmTz7bxKVM3OVQxl4X SabnVX4RMATFazfqE+8OQP2bGVy21B4= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MYHn5HiN; spf=pass (imf10.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659645601; a=rsa-sha256; cv=none; b=thHPwFDqICKNC9F+0j5nvFeIwrIzyuO2YhkkF4EO32DqrzAMfxNberegrae9ZDsNdm7Udg h6K/fBy4hC6Cb8hREefn6ueLpXzTwGy4GOVELdNsTdNVtSkPC0S4VUZj/DXJhKQf3gulNm GQRINQbRmEcF00XMGu1GtBVOzsQQ7CY= X-Rspam-User: Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MYHn5HiN; spf=pass (imf10.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: D25EAC0129 X-Stat-Signature: m933x6hbym4tcyej8yfk73gnrudq45nj X-HE-Tag: 1659645600-230693 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When page migration happens, we always ignore the young/dirty bit settings in the old pgtable, and marking the page as old in the new page table using either pte_mkold() or pmd_mkold(), and keeping the pte clean. That's fine from functional-wise, but that's not friendly to page reclaim because the moving page can be actively accessed within the procedure. Not to mention hardware setting the young bit can bring quite some overhead on some systems, e.g. x86_64 needs a few hundreds nanoseconds to set the bit. The same slowdown problem to dirty bits when the memory is first written after page migration happened. Actually we can easily remember the A/D bit configuration and recover the information after the page is migrated. To achieve it, define a new set of bits in the migration swap offset field to cache the A/D bits for old pte. Then when removing/recovering the migration entry, we can recover the A/D bits even if the page changed. One thing to mention is that here we used max_swapfile_size() to detect how many swp offset bits we have, and we'll only enable this feature if we know the swp offset can be big enough to store both the PFN value and the young bit. Otherwise the A/D bits are dropped like before. Signed-off-by: Peter Xu --- include/linux/swapops.h | 91 +++++++++++++++++++++++++++++++++++++++++ mm/huge_memory.c | 26 +++++++++++- mm/migrate.c | 6 ++- mm/migrate_device.c | 4 ++ mm/rmap.c | 5 ++- 5 files changed, 128 insertions(+), 4 deletions(-) diff --git a/include/linux/swapops.h b/include/linux/swapops.h index 1d17e4bb3d2f..34aa448ac6ee 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -8,6 +8,8 @@ #ifdef CONFIG_MMU +#include + /* * swapcache pages are stored in the swapper_space radix tree. We want to * get good packing density in that tree, so the index should be dense in @@ -35,6 +37,24 @@ #endif #define SWP_PFN_MASK ((1UL << SWP_PFN_BITS) - 1) +/** + * Migration swap entry specific bitfield definitions. + * + * @SWP_MIG_YOUNG_BIT: Whether the page used to have young bit set + * @SWP_MIG_DIRTY_BIT: Whether the page used to have dirty bit set + * + * Note: these bits will be stored in migration entries iff there're enough + * free bits in arch specific swp offset. By default we'll ignore A/D bits + * when migrating a page. Please refer to migration_entry_supports_ad() + * for more information. + */ +#define SWP_MIG_YOUNG_BIT (SWP_PFN_BITS) +#define SWP_MIG_DIRTY_BIT (SWP_PFN_BITS + 1) +#define SWP_MIG_TOTAL_BITS (SWP_PFN_BITS + 2) + +#define SWP_MIG_YOUNG (1UL << SWP_MIG_YOUNG_BIT) +#define SWP_MIG_DIRTY (1UL << SWP_MIG_DIRTY_BIT) + static inline bool is_pfn_swap_entry(swp_entry_t entry); /* Clear all flags but only keep swp_entry_t related information */ @@ -265,6 +285,57 @@ static inline swp_entry_t make_writable_migration_entry(pgoff_t offset) return swp_entry(SWP_MIGRATION_WRITE, offset); } +/* + * Returns whether the host has large enough swap offset field to support + * carrying over pgtable A/D bits for page migrations. The result is + * pretty much arch specific. + */ +static inline bool migration_entry_supports_ad(void) +{ + /* + * max_swapfile_size() returns the max supported swp-offset plus 1. + * We can support the migration A/D bits iff the pfn swap entry has + * the offset large enough to cover all of them (PFN, A & D bits). + */ +#ifdef CONFIG_SWAP + return max_swapfile_size() >= (1UL << SWP_MIG_TOTAL_BITS); +#else + return false; +#endif +} + +static inline swp_entry_t make_migration_entry_young(swp_entry_t entry) +{ + if (migration_entry_supports_ad()) + return swp_entry(swp_type(entry), + swp_offset(entry) | SWP_MIG_YOUNG); + return entry; +} + +static inline bool is_migration_entry_young(swp_entry_t entry) +{ + if (migration_entry_supports_ad()) + return swp_offset(entry) & SWP_MIG_YOUNG; + /* Keep the old behavior of aging page after migration */ + return false; +} + +static inline swp_entry_t make_migration_entry_dirty(swp_entry_t entry) +{ + if (migration_entry_supports_ad()) + return swp_entry(swp_type(entry), + swp_offset(entry) | SWP_MIG_DIRTY); + return entry; +} + +static inline bool is_migration_entry_dirty(swp_entry_t entry) +{ + if (migration_entry_supports_ad()) + return swp_offset(entry) & SWP_MIG_YOUNG_BIT; + /* Keep the old behavior of clean page after migration */ + return false; +} + extern void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, spinlock_t *ptl); extern void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, @@ -311,6 +382,26 @@ static inline int is_readable_migration_entry(swp_entry_t entry) return 0; } +static inline swp_entry_t make_migration_entry_young(swp_entry_t entry) +{ + return entry; +} + +static inline bool is_migration_entry_young(swp_entry_t entry) +{ + return false; +} + +static inline swp_entry_t make_migration_entry_dirty(swp_entry_t entry) +{ + return entry; +} + +static inline bool is_migration_entry_dirty(swp_entry_t entry) +{ + return false; +} + #endif typedef unsigned long pte_marker; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 29e3628687a6..7d79db4bd76a 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2005,6 +2005,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, pmd_t old_pmd, _pmd; bool young, write, soft_dirty, pmd_migration = false, uffd_wp = false; bool anon_exclusive = false; + bool dirty = false; unsigned long addr; int i; @@ -2088,7 +2089,8 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, write = is_writable_migration_entry(entry); if (PageAnon(page)) anon_exclusive = is_readable_exclusive_migration_entry(entry); - young = false; + young = is_migration_entry_young(entry); + dirty = is_migration_entry_dirty(entry); soft_dirty = pmd_swp_soft_dirty(old_pmd); uffd_wp = pmd_swp_uffd_wp(old_pmd); } else { @@ -2097,6 +2099,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, SetPageDirty(page); write = pmd_write(old_pmd); young = pmd_young(old_pmd); + dirty = pmd_dirty(old_pmd); soft_dirty = pmd_soft_dirty(old_pmd); uffd_wp = pmd_uffd_wp(old_pmd); @@ -2146,6 +2149,10 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, else swp_entry = make_readable_migration_entry( page_to_pfn(page + i)); + if (young) + swp_entry = make_migration_entry_young(swp_entry); + if (dirty) + swp_entry = make_migration_entry_dirty(swp_entry); entry = swp_entry_to_pte(swp_entry); if (soft_dirty) entry = pte_swp_mksoft_dirty(entry); @@ -2160,6 +2167,12 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = pte_wrprotect(entry); if (!young) entry = pte_mkold(entry); + if (dirty) + /* + * NOTE: this may contains setting soft + * dirty too on some archs like x86. + */ + entry = pte_mkdirty(entry); if (soft_dirty) entry = pte_mksoft_dirty(entry); if (uffd_wp) @@ -3148,6 +3161,10 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, entry = make_readable_exclusive_migration_entry(page_to_pfn(page)); else entry = make_readable_migration_entry(page_to_pfn(page)); + if (pmd_young(pmdval)) + entry = make_migration_entry_young(entry); + if (pmd_dirty(pmdval)) + entry = make_migration_entry_dirty(entry); pmdswp = swp_entry_to_pmd(entry); if (pmd_soft_dirty(pmdval)) pmdswp = pmd_swp_mksoft_dirty(pmdswp); @@ -3173,13 +3190,18 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) entry = pmd_to_swp_entry(*pvmw->pmd); get_page(new); - pmde = pmd_mkold(mk_huge_pmd(new, READ_ONCE(vma->vm_page_prot))); + pmde = mk_huge_pmd(new, READ_ONCE(vma->vm_page_prot)); if (pmd_swp_soft_dirty(*pvmw->pmd)) pmde = pmd_mksoft_dirty(pmde); if (is_writable_migration_entry(entry)) pmde = maybe_pmd_mkwrite(pmde, vma); if (pmd_swp_uffd_wp(*pvmw->pmd)) pmde = pmd_wrprotect(pmd_mkuffd_wp(pmde)); + if (!is_migration_entry_young(entry)) + pmde = pmd_mkold(pmde); + if (PageDirty(new) && is_migration_entry_dirty(entry)) + /* NOTE: this may contain setting soft-dirty on some archs */ + pmde = pmd_mkdirty(pmde); if (PageAnon(new)) { rmap_t rmap_flags = RMAP_COMPOUND; diff --git a/mm/migrate.c b/mm/migrate.c index 1649270bc1a7..957e8e6ee28e 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -199,7 +199,7 @@ static bool remove_migration_pte(struct folio *folio, #endif folio_get(folio); - pte = pte_mkold(mk_pte(new, READ_ONCE(vma->vm_page_prot))); + pte = mk_pte(new, READ_ONCE(vma->vm_page_prot)); if (pte_swp_soft_dirty(*pvmw.pte)) pte = pte_mksoft_dirty(pte); @@ -207,6 +207,10 @@ static bool remove_migration_pte(struct folio *folio, * Recheck VMA as permissions can change since migration started */ entry = pte_to_swp_entry(*pvmw.pte); + if (!is_migration_entry_young(entry)) + pte = pte_mkold(pte); + if (folio_test_dirty(folio) && is_migration_entry_dirty(entry)) + pte = pte_mkdirty(pte); if (is_writable_migration_entry(entry)) pte = maybe_mkwrite(pte, vma); else if (pte_swp_uffd_wp(*pvmw.pte)) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 7feeb447e3b9..47d90df04cc6 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -221,6 +221,10 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, else entry = make_readable_migration_entry( page_to_pfn(page)); + if (pte_young(pte)) + entry = make_migration_entry_young(entry); + if (pte_dirty(pte)) + entry = make_migration_entry_dirty(entry); swp_pte = swp_entry_to_pte(entry); if (pte_present(pte)) { if (pte_soft_dirty(pte)) diff --git a/mm/rmap.c b/mm/rmap.c index af775855e58f..28aef434ea41 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2065,7 +2065,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, else entry = make_readable_migration_entry( page_to_pfn(subpage)); - + if (pte_young(pteval)) + entry = make_migration_entry_young(entry); + if (pte_dirty(pteval)) + entry = make_migration_entry_dirty(entry); swp_pte = swp_entry_to_pte(entry); if (pte_soft_dirty(pteval)) swp_pte = pte_swp_mksoft_dirty(swp_pte);