From patchwork Sun Feb 25 07:56:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kasireddy, Vivek" X-Patchwork-Id: 13570755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0919CC54E41 for ; Sun, 25 Feb 2024 08:24:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 758276B00F8; Sun, 25 Feb 2024 03:24:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 706A16B00F9; Sun, 25 Feb 2024 03:24:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 583006B00FA; Sun, 25 Feb 2024 03:24:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 46FD36B00F8 for ; Sun, 25 Feb 2024 03:24:17 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 17D7FC0289 for ; Sun, 25 Feb 2024 08:24:17 +0000 (UTC) X-FDA: 81829638954.24.EE99E5D Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) by imf21.hostedemail.com (Postfix) with ESMTP id 1CCDD1C000E for ; Sun, 25 Feb 2024 08:24:14 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=nj7L7fBQ; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf21.hostedemail.com: domain of vivek.kasireddy@intel.com designates 192.198.163.17 as permitted sender) smtp.mailfrom=vivek.kasireddy@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708849455; a=rsa-sha256; cv=none; b=rx2y3BlHX4Uyp59Uj5YV4yBgFSLXxEJfbU/8+w8N8ZTTf7tbJjEoTVmscFcCuZQAbne0v9 z/j8KAxFmPhKkT+I64FMw8N7TXwuxspwFdUUzlgRU69+7V1DQ6WSsq0ih0WHHeifbz2BHA sF3gXl4HQU6YelS2nXNfxoK2Qx5K7zU= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=nj7L7fBQ; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf21.hostedemail.com: domain of vivek.kasireddy@intel.com designates 192.198.163.17 as permitted sender) smtp.mailfrom=vivek.kasireddy@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708849455; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=M77rke642uG3IUkL73gZsVZhWQrM3CDqVbQFZ5hkz84=; b=Z7pidzljTW6FBN6xzQyH+cvQCz0I7QZyRwjZeCYjrn02qN2dJ6u1uGNOnQf4VXWNJiD6r3 21mRqKAD1lK2ruweLZTg3XH1E68ysozC92gd8qBmL3IGVXb38p5ch6qVqFcnUcLxxxZ6QM PSLV8/2Vlmw9BUWKgPR1oElWnq5QgOs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708849455; x=1740385455; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qAWI9OROZ7HrTscI+EJd/d4/as0t64bOlXO/zqAPSsI=; b=nj7L7fBQ7xE+oteS4I0hZjTdOKY/q1hfvgntuSPaSQd/Z/0vuSRqlzRl agp05txi24x1PBrJD+IzJjPpDi3GPL+BN5Nuo4LL2JrIbhyhFJvhPsBEo MoABKwKQ37faOEWFSktkb2pRLcv41o57jH9j2gYeJyqgbhCT6F4zDQusF a4bslBKWPb8jske8z0Kh0G/T1YIxfkUCHPQ+C2YK8LuuiL+DvcaPeftCk jDGlsdEri1fjxqaHK45nHQkMybRhdQzz/b7QobpthiGqnUXerrkaYUsYl /wI4xi2ewPbco0TVWrnLhMC4FgU80CFx0Hh+YO06U7Rx4R9hHTPCTyipo A==; X-IronPort-AV: E=McAfee;i="6600,9927,10994"; a="2988352" X-IronPort-AV: E=Sophos;i="6.06,183,1705392000"; d="scan'208";a="2988352" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Feb 2024 00:24:11 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,183,1705392000"; d="scan'208";a="6783246" Received: from vkasired-desk2.fm.intel.com ([10.105.128.132]) by orviesa006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Feb 2024 00:24:11 -0800 From: Vivek Kasireddy To: dri-devel@lists.freedesktop.org, linux-mm@kvack.org Cc: Vivek Kasireddy , David Hildenbrand , Matthew Wilcox , Christoph Hellwig , Jason Gunthorpe , Peter Xu Subject: [PATCH v12 2/8] mm/gup: Introduce check_and_migrate_movable_folios() Date: Sat, 24 Feb 2024 23:56:58 -0800 Message-ID: <20240225080008.1019653-3-vivek.kasireddy@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240225080008.1019653-1-vivek.kasireddy@intel.com> References: <20240225080008.1019653-1-vivek.kasireddy@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 1CCDD1C000E X-Stat-Signature: 43eo8pi7gyra3ydrnb4ap184rppthuds X-HE-Tag: 1708849454-634706 X-HE-Meta: U2FsdGVkX1866K+DlGXm2vP8P6LoIhC7tb5yK1iUbH0eG639QCLBqKNNXtR8OuK/XYK+zHXJNaM1MZF3FQtAfMWpGju3gAR0BE2VzVPtce7b/C6fQaeP25K3pyKrASBhlVnVm0zY+TKqgKNK/MmgDfCjcYiO4fEqfUin417z2mpI2Iep11BJlM8Yau/xcBPxO6GISMOE9Tqq95HmwlXzQTwH4iGTQcmv+Ko/EIjdgY2XE6TAsAUgsx2KIXejoF3HXOP3vjzh5AIYfQYWlqIjkhBWcSEFbj2DMsoe5IUb0DKhaX+JxEIz4fQfv5A54dwfS8vHw8vTO6nLj4LKVQ1hER7GS49RzlRVKUlWUfgVtKVnhl04wS1j8zEVLy/f48jSNAg9AmEd0Njc91/6fauDBh62cYd/Uhm/BkK9DnaJQFdbJpDMX8yziUEI0YJ0QTzQJt7aBWxQ0rr3QV2Zn0RURsXngnXq6Kz207z1A81w+qvtevj44pK42xp2SXef//KI+GTunEjKFYKoK4y6DDLWZ9M25PviIg12rpzWPkxBoKhn0jhg+N+9iJ+d6dJakTeTmwj6XODTHcaMHZIQOehii5GMLb6H8WCUZ/a5/gRXm9NjspJJsqRDJTgH3CgRprA4fJfGQlZ/0fvdTwthlyS+dD/jWIvTGfV0QNoQsKv78CoDxXqVd135CSebohXnA7T/mT976rFa9BC7HRqoEPAckoFB5eNfwEQdwiniF8C74U7trF8wqVy2lHpkeNBrm4TSHU+0+D2eSTWoNPmmor/17lfyHAq4OK8iEC4RJ5NkRGEDEUqKqvnUZH5bpgY/XscRgIfDzWAYCLc1epNg9dFeSHcGHgBfaP+O9yyrypZM8rhsuDo8sB3mtpULDnhwSQ1aJv7QTAj2Cg27JhM4ga6JHIPA8IH9syXxYzYYvrsnoErWD1c8b3BN6KtZpwYjRycORY2gEbTdK9e2pf+ptU0 ESLr4ZWv DYu9AFsRaUsvIuekLl/0QT2eIKBgJawA+ntUMrcpiel5ijDV2raSEi/kpEej42m3jg0kCFI8cv0zRu+D3Ga9mCfkcQ/n9vxD2t5uTpZCRX8K/OOLQr+4f8RMugL5Hku+pkfcLz4DYvzSEiiGV/ZO0I1l1OY6EWfZtyEitA8dWmMM+P5zf856njNBiViZxc/bNgwwy//UHsdOVlg/7+ySRbD+KJc8pHHU4gkXMkA7sNaSCWKCnkgNi+A0LNHWzN4P5K/Km4nSoe5vTobjCDdQ/x1J0UNtfTIvBEWaoFKVfK9dRVuJ2zuZNAXp9lA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This helper is the folio equivalent of check_and_migrate_movable_pages(). Therefore, all the rules that apply to check_and_migrate_movable_pages() also apply to this one as well. Currently, this helper is only used by memfd_pin_folios(). This patch also includes changes to rename and convert the internal functions collect_longterm_unpinnable_pages() and migrate_longterm_unpinnable_pages() to work on folios. Since they are also used by check_and_migrate_movable_pages(), a temporary array is used to collect and share the folios with these functions. Cc: David Hildenbrand Cc: Matthew Wilcox Cc: Christoph Hellwig Cc: Jason Gunthorpe Cc: Peter Xu Suggested-by: David Hildenbrand Signed-off-by: Vivek Kasireddy --- mm/gup.c | 129 +++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 92 insertions(+), 37 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 0a45eda6aaeb..1410af954a4e 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2099,20 +2099,24 @@ struct page *get_dump_page(unsigned long addr) #ifdef CONFIG_MIGRATION /* - * Returns the number of collected pages. Return value is always >= 0. + * Returns the number of collected folios. Return value is always >= 0. */ -static unsigned long collect_longterm_unpinnable_pages( - struct list_head *movable_page_list, - unsigned long nr_pages, +static unsigned long collect_longterm_unpinnable_folios( + struct list_head *movable_folio_list, + unsigned long nr_folios, + struct folio **folios, struct page **pages) { unsigned long i, collected = 0; struct folio *prev_folio = NULL; bool drain_allow = true; + struct folio *folio; - for (i = 0; i < nr_pages; i++) { - struct folio *folio = page_folio(pages[i]); + for (i = 0; i < nr_folios; i++) { + if (pages) + folios[i] = page_folio(pages[i]); + folio = folios[i]; if (folio == prev_folio) continue; prev_folio = folio; @@ -2126,7 +2130,7 @@ static unsigned long collect_longterm_unpinnable_pages( continue; if (folio_test_hugetlb(folio)) { - isolate_hugetlb(folio, movable_page_list); + isolate_hugetlb(folio, movable_folio_list); continue; } @@ -2138,7 +2142,7 @@ static unsigned long collect_longterm_unpinnable_pages( if (!folio_isolate_lru(folio)) continue; - list_add_tail(&folio->lru, movable_page_list); + list_add_tail(&folio->lru, movable_folio_list); node_stat_mod_folio(folio, NR_ISOLATED_ANON + folio_is_file_lru(folio), folio_nr_pages(folio)); @@ -2148,27 +2152,28 @@ static unsigned long collect_longterm_unpinnable_pages( } /* - * Unpins all pages and migrates device coherent pages and movable_page_list. - * Returns -EAGAIN if all pages were successfully migrated or -errno for failure - * (or partial success). + * Unpins all folios and migrates device coherent folios and movable_folio_list. + * Returns -EAGAIN if all folios were successfully migrated or -errno for + * failure (or partial success). */ -static int migrate_longterm_unpinnable_pages( - struct list_head *movable_page_list, - unsigned long nr_pages, - struct page **pages) +static int migrate_longterm_unpinnable_folios( + struct list_head *movable_folio_list, + unsigned long nr_folios, + struct folio **folios) { int ret; unsigned long i; - for (i = 0; i < nr_pages; i++) { - struct folio *folio = page_folio(pages[i]); + for (i = 0; i < nr_folios; i++) { + struct folio *folio = folios[i]; if (folio_is_device_coherent(folio)) { /* - * Migration will fail if the page is pinned, so convert - * the pin on the source page to a normal reference. + * Migration will fail if the folio is pinned, so + * convert the pin on the source folio to a normal + * reference. */ - pages[i] = NULL; + folios[i] = NULL; folio_get(folio); gup_put_folio(folio, 1, FOLL_PIN); @@ -2181,23 +2186,23 @@ static int migrate_longterm_unpinnable_pages( } /* - * We can't migrate pages with unexpected references, so drop + * We can't migrate folios with unexpected references, so drop * the reference obtained by __get_user_pages_locked(). - * Migrating pages have been added to movable_page_list after + * Migrating folios have been added to movable_folio_list after * calling folio_isolate_lru() which takes a reference so the - * page won't be freed if it's migrating. + * folio won't be freed if it's migrating. */ - unpin_user_page(pages[i]); - pages[i] = NULL; + unpin_folio(folios[i]); + folios[i] = NULL; } - if (!list_empty(movable_page_list)) { + if (!list_empty(movable_folio_list)) { struct migration_target_control mtc = { .nid = NUMA_NO_NODE, .gfp_mask = GFP_USER | __GFP_NOWARN, }; - if (migrate_pages(movable_page_list, alloc_migration_target, + if (migrate_pages(movable_folio_list, alloc_migration_target, NULL, (unsigned long)&mtc, MIGRATE_SYNC, MR_LONGTERM_PIN, NULL)) { ret = -ENOMEM; @@ -2205,15 +2210,15 @@ static int migrate_longterm_unpinnable_pages( } } - putback_movable_pages(movable_page_list); + putback_movable_pages(movable_folio_list); return -EAGAIN; err: - for (i = 0; i < nr_pages; i++) - if (pages[i]) - unpin_user_page(pages[i]); - putback_movable_pages(movable_page_list); + for (i = 0; i < nr_folios; i++) + if (folios[i]) + unpin_folio(folios[i]); + putback_movable_pages(movable_folio_list); return ret; } @@ -2237,16 +2242,60 @@ static int migrate_longterm_unpinnable_pages( static long check_and_migrate_movable_pages(unsigned long nr_pages, struct page **pages) { + unsigned long nr_folios = nr_pages; unsigned long collected; - LIST_HEAD(movable_page_list); + LIST_HEAD(movable_folio_list); + struct folio **folios; + long ret; - collected = collect_longterm_unpinnable_pages(&movable_page_list, - nr_pages, pages); + folios = kmalloc_array(nr_folios, sizeof(*folios), GFP_KERNEL); + if (!folios) + return -ENOMEM; + + collected = collect_longterm_unpinnable_folios(&movable_folio_list, + nr_folios, folios, + pages); + if (!collected) { + kfree(folios); + return 0; + } + + ret = migrate_longterm_unpinnable_folios(&movable_folio_list, + nr_folios, folios); + kfree(folios); + return ret; +} + +/* + * Check whether all folios are *allowed* to be pinned. Rather confusingly, all + * folios in the range are required to be pinned via FOLL_PIN, before calling + * this routine. + * + * If any folios in the range are not allowed to be pinned, then this routine + * will migrate those folios away, unpin all the folios in the range and return + * -EAGAIN. The caller should re-pin the entire range with FOLL_PIN and then + * call this routine again. + * + * If an error other than -EAGAIN occurs, this indicates a migration failure. + * The caller should give up, and propagate the error back up the call stack. + * + * If everything is OK and all folios in the range are allowed to be pinned, + * then this routine leaves all folios pinned and returns zero for success. + */ +static long check_and_migrate_movable_folios(unsigned long nr_folios, + struct folio **folios) +{ + unsigned long collected; + LIST_HEAD(movable_folio_list); + + collected = collect_longterm_unpinnable_folios(&movable_folio_list, + nr_folios, folios, + NULL); if (!collected) return 0; - return migrate_longterm_unpinnable_pages(&movable_page_list, nr_pages, - pages); + return migrate_longterm_unpinnable_folios(&movable_folio_list, + nr_folios, folios); } #else static long check_and_migrate_movable_pages(unsigned long nr_pages, @@ -2254,6 +2303,12 @@ static long check_and_migrate_movable_pages(unsigned long nr_pages, { return 0; } + +static long check_and_migrate_movable_folios(unsigned long nr_folios, + struct folio **folios) +{ + return 0; +} #endif /* CONFIG_MIGRATION */ /*