From patchwork Sat Feb 18 00:28:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13145396 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC69FC05027 for ; Sat, 18 Feb 2023 00:29:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BD44828001A; Fri, 17 Feb 2023 19:29:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B3654280002; Fri, 17 Feb 2023 19:29:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9FE6828001A; Fri, 17 Feb 2023 19:29:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8FFF0280002 for ; Fri, 17 Feb 2023 19:29:16 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6F072405E8 for ; Sat, 18 Feb 2023 00:29:16 +0000 (UTC) X-FDA: 80478528312.21.E6BEB14 Received: from mail-vk1-f201.google.com (mail-vk1-f201.google.com [209.85.221.201]) by imf13.hostedemail.com (Postfix) with ESMTP id B504120007 for ; Sat, 18 Feb 2023 00:29:14 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="dX7/40KB"; spf=pass (imf13.hostedemail.com: domain of 32RvwYwoKCPYhrfmsefrmlemmejc.amkjglsv-kkitYai.mpe@flex--jthoughton.bounces.google.com designates 209.85.221.201 as permitted sender) smtp.mailfrom=32RvwYwoKCPYhrfmsefrmlemmejc.amkjglsv-kkitYai.mpe@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676680154; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6R+bM0KtFeTkcEJ8SgP6306LOQusudAcElSasV+tkuA=; b=6akXmLsk7TaSwa15JC/h2c0u/BImGRBkkCRRSNzQxeMTHd1G5vyg1TAjsrdkqpL0lOm+Q1 bcgfDK1OZXjJg6tlFqe+a8zRZYeOFfKa14A7HcrbPXOOxT1Z/mghiMQhrjsXfp7w+gh9mF fht5Iln3Hbrzy0TGQpSZr1/oPZxymQk= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="dX7/40KB"; spf=pass (imf13.hostedemail.com: domain of 32RvwYwoKCPYhrfmsefrmlemmejc.amkjglsv-kkitYai.mpe@flex--jthoughton.bounces.google.com designates 209.85.221.201 as permitted sender) smtp.mailfrom=32RvwYwoKCPYhrfmsefrmlemmejc.amkjglsv-kkitYai.mpe@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676680154; a=rsa-sha256; cv=none; b=fR9WGsU9LjyuhGE/wX88zaZncCA8YaT1YMJt+eC66gq45UFVzfSjuuN/rAzDYa0pI0SWm5 Q1pjG0h9xH0jPi5Od/40Wmpp249rPxSjqdo8ZjPVUE/1UiBIPQ9zEJbBnbhRuEMRQSZNVl u1g17D8ZcsVRYbXdC6zpDb3bnHNpA2o= Received: by mail-vk1-f201.google.com with SMTP id g1-20020ac5c5c1000000b00401b81d313bso828604vkl.6 for ; Fri, 17 Feb 2023 16:29:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1676680154; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6R+bM0KtFeTkcEJ8SgP6306LOQusudAcElSasV+tkuA=; b=dX7/40KBqq/CpQhiOpkAQIzPNluCgT+pqoGZRKlN9aa8QluovsQruDCu/wh6+Xi7HL 6kLXZgl+5XKDk7FsMYGYtr2yyFU9erZiSo7V2dyAwIFPJO5DAOooAQa2F7w+5zW0Hqdu QubYMhLcbqNzMIVdTubD/VkEI82Q/CwfufnycIJ0sLbdRiI6AathfyY49TSn1Ly/A6+9 +4K1wirIYyv3shJT02BYKt93mVx2ZH0jSU1+ZgamxSym8xZo1mO/BLj8U+ZTd86LEOwb MxBJulRTBsOmTowXosHxX9g5jLuhbpjY1XM7qMG0aeLiOSSYz9qPT0kULRTK/B6ula9l hMbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1676680154; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6R+bM0KtFeTkcEJ8SgP6306LOQusudAcElSasV+tkuA=; b=qMV8XtYsVdybPM09+q/Zsc/J5Rto+5KN9/lBDGHaH6VJPC7m/QkSlw44uD2v5Uvgh3 XcXwP2fd8XAv8ICrxX1ol64/k8b0Xykgeeaqsyk5P5uhctyDOXgf3vyXt4nzYy9OGUAt gdZZfOCub7cnTmPEg+g7737YzI3jdmzVMRxb7u/j5N0BWZ7Wr00FhALdftOeWtVhUJBY ynXE2MB50uup7hn/TlBIehLciczB3IZEHpUYPvBI8I2uoGkIOJTylESGHo/j+6R32hgP 4+NvlsC6oE4/5TszGHRsIJVsEkfZxwtsbZCyvmr7hWKbp4K9nKmmBj5C1BZkXwtv4VuR Sqdg== X-Gm-Message-State: AO0yUKVCXHROXicFsksxnNWzaCcMgNLIWe8c9gs+OH3r0yUhQ/vKSWwP d/309f9njHh+4f4sIMYO1lBhNP7uG1Toc2dz X-Google-Smtp-Source: AK7set8ydXZ+Dciv1+JvuwKHd4m9xN+HXoJNqOMVwMnv6hYgnClrworMO3PvsXyUer/h8BcS4rjrDLd27tp3RFfQ X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a1f:a041:0:b0:401:7fe9:ff7f with SMTP id j62-20020a1fa041000000b004017fe9ff7fmr213533vke.5.1676680153948; Fri, 17 Feb 2023 16:29:13 -0800 (PST) Date: Sat, 18 Feb 2023 00:28:03 +0000 In-Reply-To: <20230218002819.1486479-1-jthoughton@google.com> Mime-Version: 1.0 References: <20230218002819.1486479-1-jthoughton@google.com> X-Mailer: git-send-email 2.39.2.637.g21b0678d19-goog Message-ID: <20230218002819.1486479-31-jthoughton@google.com> Subject: [PATCH v2 30/46] hugetlb: add high-granularity migration support From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu , Andrew Morton Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , "Zach O'Keefe" , Manish Mishra , Naoya Horiguchi , "Dr . David Alan Gilbert" , "Matthew Wilcox (Oracle)" , Vlastimil Babka , Baolin Wang , Miaohe Lin , Yang Shi , Frank van der Linden , Jiaqi Yan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton X-Rspamd-Queue-Id: B504120007 X-Stat-Signature: 5s7bu8tacfsesfnugz4cpwr9gcjobwpx X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1676680154-170761 X-HE-Meta: U2FsdGVkX1/dp4Yj/Q76jIsdLa8XbeoEOmdRqDmYtpD5iO4fhBJ+jTEx1TYQnPakHJDJrSct8TlfOnvIMTRkasmduAt+wygtuOF6n8PdZhqPCxhWjJ8oA1DZrrGpxhqSiOw8ty37W52Y3bnEUHqEkKb7UxqXEJCarJekC+JT6iDlGiFfiIaQbkmqsE4uCui53BaTY44P2IYSMEUPzVpO85/PSho3wNX5g8r9HJUYFv0fRlPg8EbC/sjbW7OHOhuXolmT1yAECsupMp9xtU5DiCXvA+RpIcpRVCNnRguMoFHiBzDI7rjBbcD3ckgt5gdNaImxyAfqqQMorH/TR/ID2L/6wL+KL/ONJmSv6IZ5eF310z2UYzGTY9N9Xnjor5A14TdfJGyDXU2/CcMMXSBABNC/TEfEf7F0kqpn5XrTt4HFqyr7p0TuZnMRiMgwN1HSKHm0oUBEj1I0DKpTQ+FlH2iOLgsyMBq/te0oDUsjFNCZRW49nJP7HPmHr24JZKA4j67DJiqLkKx86YodYH23GblV/oA9V/o80pdeZz2thLNd7jiVc1FSsSMZstRyex3lqYqTbgxNe61r+eDNpm7Shb1PWKJcSIzhDqOwpt3FmPrkqSuAk6hm1sVtO/GILvg/iPrQrDVzY8E58M7PBhYpzFMYfXL4qpYyEp0x+PIsqftUc+3jhoN3B/IFih0qeAQvFv0Rts9MxlqL1aR6Bk7hTw2M28am9NtfRrqA99jucpEKKbqB/U7t77PW+38PC4EQTiyektPw2sMU4WjryTnX24cko4ke6ssEJc8lcY9FH5Nch969lGJg02sKy/R4nptogNq9DycRHytMk3o01HTPJa1eEpBVkj9KC/9LJD31+bKt94a+Jczg0Z6xeX6y/BD+psReRf+YuO+PrExXYhW6qIVPFNro1FyRaIJaxNVP28MQmvF6WjwW1rfw+5jrDIuIE6plIzaZT+iUEcgygss FySu1DGz np0BObbZWcKs1lBZF7LNSPBjkUdHN1vFQn892j2lF4mhU/Q0u9pyDzOLO4GNDYjgMRmA77IXLDI5v8utyODE8Gwu1qim/H3F9/LkyVkeTc7gjv0mznd/MXRteOyxxctBrHUGO1lt0XuFqxCOS/3kxUw00cSNCxgQbnF2wQ+VAONo4XJP+gY/2s5m/qon10kGorMkIsQRaXIIDjHTf46REocr/cK8rY9OuGRAtVQXOe6tSRYOidVuL3KbUox6PlQ+MSX3em+dmc81nZaQreLVJERUHpVn6c4788M4JvChzb1QqGpNWlCC335akA4nM9S2clJS6l4wp69BXjgzg+XaDKk+xGla+MC/psAQEvJA6jh5z3k+2tUOichn197tQX+Edun6ye5QUcxtEk0jJZTAVQHqB+FsI0UyORhpk31944ZdOZtFmf5Kr6Iwb1AMrfa3ZJut2gCdXFsjsl2UNtHjYIQk+q0Vi5D+8AQcPzzEYXXSRREuDU0L33fxnqC6IqX/ztHcEJ+jPOL9Sh1eYhySCdklJboVFXOqiOmusotAuyuPRBzKUfK/ie0q3GDZmjTfzbHlavxEU4reK6xX6FtWVjY8qV464WlbyDg2F1Ky9FOMBGOGF8Vg+6+GLYhImUVgzCUImXf4LDyizAk58Ll3YyrrGCPuTr1ttzvlf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To prevent queueing a hugepage for migration multiple times, we use last_folio to keep track of the last page we saw in queue_pages_hugetlb, and if the page we're looking at is last_folio, then we skip it. For the non-hugetlb cases, last_folio, although unused, is still updated so that it has a consistent meaning with the hugetlb case. Signed-off-by: James Houghton diff --git a/include/linux/swapops.h b/include/linux/swapops.h index 3a451b7afcb3..6ef80763e629 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -68,6 +68,8 @@ static inline bool is_pfn_swap_entry(swp_entry_t entry); +struct hugetlb_pte; + /* Clear all flags but only keep swp_entry_t related information */ static inline pte_t pte_swp_clear_flags(pte_t pte) { @@ -339,7 +341,8 @@ extern void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, #ifdef CONFIG_HUGETLB_PAGE extern void __migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *ptep, spinlock_t *ptl); -extern void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte); +extern void migration_entry_wait_huge(struct vm_area_struct *vma, + struct hugetlb_pte *hpte); #endif /* CONFIG_HUGETLB_PAGE */ #else /* CONFIG_MIGRATION */ static inline swp_entry_t make_readable_migration_entry(pgoff_t offset) @@ -369,7 +372,8 @@ static inline void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, #ifdef CONFIG_HUGETLB_PAGE static inline void __migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *ptep, spinlock_t *ptl) { } -static inline void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte) { } +static inline void migration_entry_wait_huge(struct vm_area_struct *vma, + struct hugetlb_pte *hpte) { } #endif /* CONFIG_HUGETLB_PAGE */ static inline int is_writable_migration_entry(swp_entry_t entry) { diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 86cd51beb02c..39f541b4a0a8 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6418,7 +6418,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * be released there. */ mutex_unlock(&hugetlb_fault_mutex_table[hash]); - migration_entry_wait_huge(vma, hpte.ptep); + migration_entry_wait_huge(vma, &hpte); return 0; } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry))) ret = VM_FAULT_HWPOISON_LARGE | diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 0f91be88392b..43e210181cce 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -424,6 +424,7 @@ struct queue_pages { unsigned long start; unsigned long end; struct vm_area_struct *first; + struct folio *last_folio; }; /* @@ -475,6 +476,7 @@ static int queue_folios_pmd(pmd_t *pmd, spinlock_t *ptl, unsigned long addr, flags = qp->flags; /* go to folio migration */ if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) { + qp->last_folio = folio; if (!vma_migratable(walk->vma) || migrate_folio_add(folio, qp->pagelist, flags)) { ret = 1; @@ -539,6 +541,8 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, break; } + qp->last_folio = folio; + /* * Do not abort immediately since there may be * temporary off LRU pages in the range. Still @@ -570,15 +574,22 @@ static int queue_folios_hugetlb(struct hugetlb_pte *hpte, spinlock_t *ptl; pte_t entry; - /* We don't migrate high-granularity HugeTLB mappings for now. */ - if (hugetlb_hgm_enabled(walk->vma)) - return -EINVAL; - ptl = hugetlb_pte_lock(hpte); entry = huge_ptep_get(hpte->ptep); if (!pte_present(entry)) goto unlock; - folio = pfn_folio(pte_pfn(entry)); + + if (!hugetlb_pte_present_leaf(hpte, entry)) { + ret = -EAGAIN; + goto unlock; + } + + folio = page_folio(pte_page(entry)); + + /* We already queued this page with another high-granularity PTE. */ + if (folio == qp->last_folio) + goto unlock; + if (!queue_folio_required(folio, qp)) goto unlock; @@ -747,6 +758,7 @@ queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, .start = start, .end = end, .first = NULL, + .last_folio = NULL, }; err = walk_page_range(mm, start, end, &queue_pages_walk_ops, &qp); diff --git a/mm/migrate.c b/mm/migrate.c index 616afcc40fdc..b26169990532 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -196,6 +196,9 @@ static bool remove_migration_pte(struct folio *folio, /* pgoff is invalid for ksm pages, but they are never large */ if (folio_test_large(folio) && !folio_test_hugetlb(folio)) idx = linear_page_index(vma, pvmw.address) - pvmw.pgoff; + else if (folio_test_hugetlb(folio)) + idx = (pvmw.address & ~huge_page_mask(hstate_vma(vma)))/ + PAGE_SIZE; new = folio_page(folio, idx); #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION @@ -247,14 +250,16 @@ static bool remove_migration_pte(struct folio *folio, #ifdef CONFIG_HUGETLB_PAGE if (folio_test_hugetlb(folio)) { + struct page *hpage = folio_page(folio, 0); unsigned int shift = pvmw.pte_order + PAGE_SHIFT; pte = arch_make_huge_pte(pte, shift, vma->vm_flags); if (folio_test_anon(folio)) - hugepage_add_anon_rmap(new, vma, pvmw.address, + hugepage_add_anon_rmap(hpage, vma, pvmw.address, rmap_flags); else - page_add_file_rmap(new, vma, true); + hugetlb_add_file_rmap(new, shift, + hstate_vma(vma), vma); set_huge_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); } else #endif @@ -270,7 +275,7 @@ static bool remove_migration_pte(struct folio *folio, mlock_drain_local(); trace_remove_migration_pte(pvmw.address, pte_val(pte), - compound_order(new)); + pvmw.pte_order); /* No need to invalidate - it was non-present before */ update_mmu_cache(vma, pvmw.address, pvmw.pte); @@ -361,12 +366,10 @@ void __migration_entry_wait_huge(struct vm_area_struct *vma, } } -void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte) +void migration_entry_wait_huge(struct vm_area_struct *vma, + struct hugetlb_pte *hpte) { - spinlock_t *ptl = huge_pte_lockptr(huge_page_shift(hstate_vma(vma)), - vma->vm_mm, pte); - - __migration_entry_wait_huge(vma, pte, ptl); + __migration_entry_wait_huge(vma, hpte->ptep, hpte->ptl); } #endif