From patchwork Fri Aug 2 15:55:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13751721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C96FC52D6F for ; Fri, 2 Aug 2024 15:56:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2675E6B0093; Fri, 2 Aug 2024 11:56:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 218C46B0095; Fri, 2 Aug 2024 11:56:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0B98B6B0096; Fri, 2 Aug 2024 11:56:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id DDE5B6B0093 for ; Fri, 2 Aug 2024 11:56:08 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9D16E12133E for ; Fri, 2 Aug 2024 15:56:08 +0000 (UTC) X-FDA: 82407756816.29.03D436E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id C5995C0026 for ; Fri, 2 Aug 2024 15:56:06 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JngJUdyk; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722614121; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WGUlr9yCrIjZfp2Jios2/QtrJcjV6+nMbt6NumQJE9A=; b=DQCISnXvUrAOvoVb+tdnJYgeJrlNvuFTv3mS17eFhku6FMeEWUYpCZywVbat5IYyd1kxlj l0nUzOl/J5kGg89y6BNgvUykLUkQcDxuP0WyrVPWTnafAsZllY+b4+igoAArbJ5Hpo2q8g hg45XzHsfC8UCT0405mAwfl4B+9ccSY= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JngJUdyk; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722614121; a=rsa-sha256; cv=none; b=tQ1Ia0tAYmAdSA04tiiSR7yhtZjil1YkSFr3UdDeqRNCf09/DN3pMsUgH8MMx7t3d139Y/ NcjEpmnyYJjs0AOGpcAG4mM4I7ef2WKn+JCbQcJkqlXppNVBRBSCjJ/GdHBVa+AEMV+Gv1 iOLWRtPLbgok2rrM9ZGJnCDq4a0TRPY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1722614166; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WGUlr9yCrIjZfp2Jios2/QtrJcjV6+nMbt6NumQJE9A=; b=JngJUdykOhU5j/7yrQJym9Rn7lu28pSgWOBSKw3LxVFZv5THS8XmMWFbe3Aa/fmre3VwkU LJQH6tjPE5xpasvziFILZXJhGC+7cT5/A54+wpn8Snfsdp4WZJZLIpO02z0Jnwh7cGBadF cvlHB2FoEv4c/Q4jRY7H/ah3PxUQSCE= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-8-GL6qdjBaOLKZo_tI8pVNMA-1; Fri, 02 Aug 2024 11:56:03 -0400 X-MC-Unique: GL6qdjBaOLKZo_tI8pVNMA-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 9722A1955D4D; Fri, 2 Aug 2024 15:55:59 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.113]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 863C5300018D; Fri, 2 Aug 2024 15:55:53 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-s390@vger.kernel.org, linux-fsdevel@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Jonathan Corbet , Christian Borntraeger , Janosch Frank , Claudio Imbrenda , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Sven Schnelle , Gerald Schaefer Subject: [PATCH v1 04/11] mm/migrate: convert add_page_for_migration() from follow_page() to folio_walk Date: Fri, 2 Aug 2024 17:55:17 +0200 Message-ID: <20240802155524.517137-5-david@redhat.com> In-Reply-To: <20240802155524.517137-1-david@redhat.com> References: <20240802155524.517137-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Stat-Signature: 3qiwxnsjri6onk85ritd1s51yytky7dm X-Rspam-User: X-Rspamd-Queue-Id: C5995C0026 X-Rspamd-Server: rspam02 X-HE-Tag: 1722614166-717222 X-HE-Meta: U2FsdGVkX1+0kPgzE4bTjhckzgIxCbWQSHhwAoJQGh5Uh529ES1OD2EsGoD7VVrVzsS30mLmNufCPMXPKNNsUEYw0ByPAQ8HzPDDQak8+6Ar9Ntd3qkhO+AeuETMzCFXPMDevtXOrrWd5Hrwwg6POjMa4oEhWdkFOTxjYhQD7qLid1e4pAfOY6qEVXmbi63Dtz+tcexdPySLLikj4PIf7tLcaEC6ux3mkYyU6o13+KcmAA08ABOuZlIsjnSZ+zJgjUuAOYw3jI7xYMtEgLKE4ELWFDSjC5rkwmqNivu+PU8x8BkZZ/xvGr7xRXAkQVi0pfH3ykB3ZuBWO59JktrNVpvA4aDlydurYpiMrfVg7L7mDAj0j1vOQl2qUWy8uUHj0IEQPtnkJuhdx+YngsjNpGya4adsX2Bn0YoY/vFzExJacmGj2cTkSsa1lsyJKHBej6n9NUI58EBFZhFsnqexteTjpftZoubKhpSdywIuGj/uFskRLFHbxfPQ41/k68DpjsOyn5zCaY6oDsmmHsZutpBmp85z0MkMqQp6zzyzRdUCjh4qd5DBvhxny2eyFoxgLyNNH0wAbL/IleAx59GlDDFQGy6ggqobfQSIo0p24k2u1McJNlL6VPCxqz/XlXra4joRhFJYyTdKBELeteWS7J4KB8rKdBcnDFXc0KXKmWWB06nGsvPELQ3An0w2SsUpg1S1dQYFatcAyGt7o6W0/+5Ih+9SYUcgokNOwgQL1D1Rq166kcZ9VMM+oUek1pUp21fc8I5uL28wwPJz1QYDTdZigDbNxkeUA+1ZuT744tMrv3ia/RdGAePu0GnBv+VPBHJ82DwU7rp82rYObsnpHm7Axb+vyfCzNM/wJHdYcoez8XRwA/6Z2IymsojHDVjVlKag7zFklRjKOxZ9n68Sx0snUpuwxOHDdSU69239PjDxpFVfOrzdskUTXGf3Klg1I5s1ix07daBE3AnoJHA wmuzgoH3 exh+VKf+hi71uo8aiSHGFCHx5spSIbvlTKkV+9adeWawu203uhMyv2YPWf3rItRTUdniopAuxwna8hSelj5YwAdT/0Ap+pXHDK2WqEUtgAtDQ404xMQVSqufDRokGA4T4FXdrq8i52+6QVaTjvRrITLrRNrWojDZDXlZsoBhqTi+HRE8Nzla61HYVtatY9rs5+QO9wh6mTd/WhT8on2mBAhsjb14ouRv2ZxMIg72FwVycD0jdadB4KY1VQ9If+5U/4GnwlR7Z19BQPoVpEpGjS9julnK3LJqHJQP5AK5gCXZpeWIbFK+Iwoj3jFGtPZv7mDVRYxc6kOTEnXNKwyvNNEYCs/YGPOykP1VN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's use folio_walk instead, so we can avoid taking a folio reference when we won't even be trying to migrate the folio and to get rid of another follow_page()/FOLL_DUMP user. Use FW_ZEROPAGE so we can return "-EFAULT" for it as documented. We now perform the folio_likely_mapped_shared() check under PTL, which is what we want: relying on the mapcount and friends after dropping the PTL does not make too much sense, as the page can get unmapped concurrently from this process. Further, we perform the folio isolation under PTL, similar to how we handle it for MADV_PAGEOUT. The possible return values for follow_page() were confusing, especially with FOLL_DUMP set. We'll handle it like documented in the man page: * -EFAULT: This is a zero page or the memory area is not mapped by the process. * -ENOENT: The page is not present. We'll keep setting -ENOENT for ZONE_DEVICE. Maybe not the right thing to do, but it likely doesn't really matter (just like for weird devmap, whereby we fake "not present"). The other errros are left as is, and match the documentation in the man page. While at it, rename add_page_for_migration() to add_folio_for_migration(). We'll lose the "secretmem" check, but that shouldn't really matter because these folios cannot ever be migrated. Should vma_migratable() refuse these VMAs? Maybe. Signed-off-by: David Hildenbrand --- mm/migrate.c | 100 +++++++++++++++++++++++---------------------------- 1 file changed, 45 insertions(+), 55 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index b5365a434ba9..e1383d9cc944 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2112,76 +2112,66 @@ static int do_move_pages_to_node(struct list_head *pagelist, int node) return err; } +static int __add_folio_for_migration(struct folio *folio, int node, + struct list_head *pagelist, bool migrate_all) +{ + if (is_zero_folio(folio) || is_huge_zero_folio(folio)) + return -EFAULT; + + if (folio_is_zone_device(folio)) + return -ENOENT; + + if (folio_nid(folio) == node) + return 0; + + if (folio_likely_mapped_shared(folio) && !migrate_all) + return -EACCES; + + if (folio_test_hugetlb(folio)) { + if (isolate_hugetlb(folio, pagelist)) + return 1; + } else if (folio_isolate_lru(folio)) { + list_add_tail(&folio->lru, pagelist); + node_stat_mod_folio(folio, + NR_ISOLATED_ANON + folio_is_file_lru(folio), + folio_nr_pages(folio)); + return 1; + } + return -EBUSY; +} + /* - * Resolves the given address to a struct page, isolates it from the LRU and + * Resolves the given address to a struct folio, isolates it from the LRU and * puts it to the given pagelist. * Returns: - * errno - if the page cannot be found/isolated + * errno - if the folio cannot be found/isolated * 0 - when it doesn't have to be migrated because it is already on the * target node * 1 - when it has been queued */ -static int add_page_for_migration(struct mm_struct *mm, const void __user *p, +static int add_folio_for_migration(struct mm_struct *mm, const void __user *p, int node, struct list_head *pagelist, bool migrate_all) { struct vm_area_struct *vma; - unsigned long addr; - struct page *page; + struct folio_walk fw; struct folio *folio; - int err; + unsigned long addr; + int err = -EFAULT; mmap_read_lock(mm); addr = (unsigned long)untagged_addr_remote(mm, p); - err = -EFAULT; vma = vma_lookup(mm, addr); - if (!vma || !vma_migratable(vma)) - goto out; - - /* FOLL_DUMP to ignore special (like zero) pages */ - page = follow_page(vma, addr, FOLL_GET | FOLL_DUMP); - - err = PTR_ERR(page); - if (IS_ERR(page)) - goto out; - - err = -ENOENT; - if (!page) - goto out; - - folio = page_folio(page); - if (folio_is_zone_device(folio)) - goto out_putfolio; - - err = 0; - if (folio_nid(folio) == node) - goto out_putfolio; - - err = -EACCES; - if (folio_likely_mapped_shared(folio) && !migrate_all) - goto out_putfolio; - - err = -EBUSY; - if (folio_test_hugetlb(folio)) { - if (isolate_hugetlb(folio, pagelist)) - err = 1; - } else { - if (!folio_isolate_lru(folio)) - goto out_putfolio; - - err = 1; - list_add_tail(&folio->lru, pagelist); - node_stat_mod_folio(folio, - NR_ISOLATED_ANON + folio_is_file_lru(folio), - folio_nr_pages(folio)); + if (vma && vma_migratable(vma)) { + folio = folio_walk_start(&fw, vma, addr, FW_ZEROPAGE); + if (folio) { + err = __add_folio_for_migration(folio, node, pagelist, + migrate_all); + folio_walk_end(&fw, vma); + } else { + err = -ENOENT; + } } -out_putfolio: - /* - * Either remove the duplicate refcount from folio_isolate_lru() - * or drop the folio ref if it was not isolated. - */ - folio_put(folio); -out: mmap_read_unlock(mm); return err; } @@ -2275,8 +2265,8 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, * Errors in the page lookup or isolation are not fatal and we simply * report them via status */ - err = add_page_for_migration(mm, p, current_node, &pagelist, - flags & MPOL_MF_MOVE_ALL); + err = add_folio_for_migration(mm, p, current_node, &pagelist, + flags & MPOL_MF_MOVE_ALL); if (err > 0) { /* The page is successfully queued for migration */