From patchwork Tue Dec 27 00:28:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13081971 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70BF8C4332F for ; Tue, 27 Dec 2022 00:29:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F3E54900002; Mon, 26 Dec 2022 19:29:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EEC14940007; Mon, 26 Dec 2022 19:29:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D8CBE900004; Mon, 26 Dec 2022 19:29:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CAA32900002 for ; Mon, 26 Dec 2022 19:29:31 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 85E84C0532 for ; Tue, 27 Dec 2022 00:29:31 +0000 (UTC) X-FDA: 80286202542.05.356A4E2 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf11.hostedemail.com (Postfix) with ESMTP id 95FE440002 for ; Tue, 27 Dec 2022 00:29:29 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=cKxwry+c; spf=pass (imf11.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672100969; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oXkqm2R/AYFSawwF3yOuv+8E/SIxEytfCh6xrgMAPBQ=; b=HGSIGUZknPDewiBFHEnZhOWZ3dV4nawCMnGuKy1HDviWb6uE6+8gzHT9RuFfDscsxUSZVU DtG0czP2chlQ+AYB7ximDqLZhSEa+GTYQf9XCrFhWf9nUGHO0qv2hkDPCBQ0TgJ+nR9GuC 3zulMNS/g/CbxKcwlNAyFhGJyieD8ME= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=cKxwry+c; spf=pass (imf11.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672100969; a=rsa-sha256; cv=none; b=dT+Jdzlgwl8bbSuXroMhGa6S7xNeOsFC97PPBav0YXjCwtFos8h9Avyl7fEkp0TIxFWvG6 zWOisWcQPikmJvkbYXML0TJI6fhv239qJu4+4Qz+omYrQP7gFCOffONuSsb6GMutz+rLon 30/vqYPjImo+f7kw/1i8JwzFP+VyHM8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672100969; x=1703636969; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=0ZlE1166uwC0o6eW3hBf1ccltW3myzWM3kgZ5n2OLo4=; b=cKxwry+cLm8VI9ohZvCIsd/+F6+V0lTrTv1l9us7A0tfVyiSzR6RwtK3 m987HpeyuRnrcftcGXVXvnCV7sXskpoPHh5UZ2v3P8cph5Mm1x6PfWmQY 3W8awnmY2AW3dmaMPcRPJGTy0vOjTErweLZzjiamCfIXbdPUamkdTS5KG bkpfOVYmJz0EaI6/PNcneWKo7qxqCUoSFQhkTQW8snFMRQ3/Gz+GbTUhO BubGUg/rWIiDkHzFwA7dzKXzYzOjTsg3c8+FIoWUoes5P4XaHw5ze/ZBF gl4rJTTnlcKs4/dE019XGuYjAvlkzlMwoK6fMhrorP1TN0sDByGfFKnTf w==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="322597206" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="322597206" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:29 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="760172186" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="760172186" Received: from yyang3-mobl1.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.212.104]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:21 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH 1/8] migrate_pages: organize stats with struct migrate_pages_stats Date: Tue, 27 Dec 2022 08:28:52 +0800 Message-Id: <20221227002859.27740-2-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221227002859.27740-1-ying.huang@intel.com> References: <20221227002859.27740-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 95FE440002 X-Rspam-User: X-Stat-Signature: ha6nrmda45gn9z6rojh4mn9zm85qypdo X-HE-Tag: 1672100969-689497 X-HE-Meta: U2FsdGVkX1/zrjApetOQnTi9I5utv8Wcmce+opShGvBY9hrlobm7/xOuI8QJJLz4OxeqqR7FYiMnAPJrJfcF4mO+KWWOoYS6jcfzLo/icXok9kAlpAeo4VgARn/Py53ugC5tcXFHmtcqa/y9FEX4rjx+H9g1idDGZ4OBd2DLX1RniWRx4XloVPsKZ+eJzeVenwWkUHINWiZjzexdjwLnqxNnVD9XFSQp2EfaTe+e/+h/4udpWjZJxLnhX2W3miCjHlCGR94Mr+FXskMv6LOm3KbiBJieeKgVmanPpMZU2x3UEivcE3ohR15GzgxBxPmHwpKh0DvLgqWhqRljRHYNNk5zo6Em2QQCX9dp13izzFGnNIXsBRXcFibSlr26np6Kp4Lfgz9KH14K2YH4vgzhhJXcNfxcCb6h1pB3dgIXAa8F8f+Lve6xdv28w7k7Yj3ta9v1cN8IlPR83Flh+yymmMFZFYn+K0280P+m9W+9EBxaXLkJesW+SfzES/T1liQF1aMKV0atFuPhYF2ZHLdHSRxp9YiOt2Eg0IKXzgIRkrVs5yz/zR9lIg6PhyA7OBKH230mxp7oDa6ElgzKOZX3vuAh3Pzdy+0JH5GsfiC10AETUyZbwERaEXdFmhaQgl2mxQjurfsaD5aG2eQzB07gJKlaORk2HkgT9AnSY7s4pkAV+wrXts9a+A2/yOMQ0HyTo+gbOvUE2UfHpPLwyTzD6NBnt0JJ0WnlXmVf6yYf7bryWT13RjzKR/vJ6ajF5kXSfd7++Vj0s5lGb+ywetTVqna2V41Zi2XFrGfDcwzm1WaEwUUYUeCRMCEcClu0gxVFFcTtnRTg+0yuxg+SXPiYm1jgPYWjpwc1P88iUt1X3YvuLk0bDYGfxyfDRd1IUJxBdW4KCUgBn5tXB6W3RqFcGIfVKvJEMNo6ky1C95fnGGZSFLnjiOaV+Z5zYVc0FfdQobbVAeOfRB4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Define struct migrate_pages_stats to organize the various statistics in migrate_pages(). This makes it easier to collect and consume the statistics in multiple functions. This will be needed in the following patches in the series. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin Reviewed-by: Alistair Popple --- mm/migrate.c | 58 +++++++++++++++++++++++++++++----------------------- 1 file changed, 32 insertions(+), 26 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index a4d3fc65085f..ec9263a33d38 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1396,6 +1396,14 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f return rc; } +struct migrate_pages_stats { + int nr_succeeded; + int nr_failed_pages; + int nr_thp_succeeded; + int nr_thp_failed; + int nr_thp_split; +}; + /* * migrate_pages - migrate the folios specified in a list, to the free folios * supplied as the target for the page migration @@ -1430,13 +1438,8 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, int large_retry = 1; int thp_retry = 1; int nr_failed = 0; - int nr_failed_pages = 0; int nr_retry_pages = 0; - int nr_succeeded = 0; - int nr_thp_succeeded = 0; int nr_large_failed = 0; - int nr_thp_failed = 0; - int nr_thp_split = 0; int pass = 0; bool is_large = false; bool is_thp = false; @@ -1446,9 +1449,11 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, LIST_HEAD(split_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); bool no_split_folio_counting = false; + struct migrate_pages_stats stats; trace_mm_migrate_pages_start(mode, reason); + memset(&stats, 0, sizeof(stats)); split_folio_migration: for (pass = 0; pass < 10 && (retry || large_retry); pass++) { retry = 0; @@ -1502,9 +1507,9 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, /* Large folio migration is unsupported */ if (is_large) { nr_large_failed++; - nr_thp_failed += is_thp; + stats.nr_thp_failed += is_thp; if (!try_split_folio(folio, &split_folios)) { - nr_thp_split += is_thp; + stats.nr_thp_split += is_thp; break; } /* Hugetlb migration is unsupported */ @@ -1512,7 +1517,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_failed++; } - nr_failed_pages += nr_pages; + stats.nr_failed_pages += nr_pages; list_move_tail(&folio->lru, &ret_folios); break; case -ENOMEM: @@ -1522,13 +1527,13 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (is_large) { nr_large_failed++; - nr_thp_failed += is_thp; + stats.nr_thp_failed += is_thp; /* Large folio NUMA faulting doesn't split to retry. */ if (!nosplit) { int ret = try_split_folio(folio, &split_folios); if (!ret) { - nr_thp_split += is_thp; + stats.nr_thp_split += is_thp; break; } else if (reason == MR_LONGTERM_PIN && ret == -EAGAIN) { @@ -1546,7 +1551,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_failed++; } - nr_failed_pages += nr_pages + nr_retry_pages; + stats.nr_failed_pages += nr_pages + nr_retry_pages; /* * There might be some split folios of fail-to-migrate large * folios left in split_folios list. Move them back to migration @@ -1556,7 +1561,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, list_splice_init(&split_folios, from); /* nr_failed isn't updated for not used */ nr_large_failed += large_retry; - nr_thp_failed += thp_retry; + stats.nr_thp_failed += thp_retry; goto out; case -EAGAIN: if (is_large) { @@ -1568,8 +1573,8 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_retry_pages += nr_pages; break; case MIGRATEPAGE_SUCCESS: - nr_succeeded += nr_pages; - nr_thp_succeeded += is_thp; + stats.nr_succeeded += nr_pages; + stats.nr_thp_succeeded += is_thp; break; default: /* @@ -1580,20 +1585,20 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (is_large) { nr_large_failed++; - nr_thp_failed += is_thp; + stats.nr_thp_failed += is_thp; } else if (!no_split_folio_counting) { nr_failed++; } - nr_failed_pages += nr_pages; + stats.nr_failed_pages += nr_pages; break; } } } nr_failed += retry; nr_large_failed += large_retry; - nr_thp_failed += thp_retry; - nr_failed_pages += nr_retry_pages; + stats.nr_thp_failed += thp_retry; + stats.nr_failed_pages += nr_retry_pages; /* * Try to migrate split folios of fail-to-migrate large folios, no * nr_failed counting in this round, since all split folios of a @@ -1626,16 +1631,17 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, if (list_empty(from)) rc = 0; - count_vm_events(PGMIGRATE_SUCCESS, nr_succeeded); - count_vm_events(PGMIGRATE_FAIL, nr_failed_pages); - count_vm_events(THP_MIGRATION_SUCCESS, nr_thp_succeeded); - count_vm_events(THP_MIGRATION_FAIL, nr_thp_failed); - count_vm_events(THP_MIGRATION_SPLIT, nr_thp_split); - trace_mm_migrate_pages(nr_succeeded, nr_failed_pages, nr_thp_succeeded, - nr_thp_failed, nr_thp_split, mode, reason); + count_vm_events(PGMIGRATE_SUCCESS, stats.nr_succeeded); + count_vm_events(PGMIGRATE_FAIL, stats.nr_failed_pages); + count_vm_events(THP_MIGRATION_SUCCESS, stats.nr_thp_succeeded); + count_vm_events(THP_MIGRATION_FAIL, stats.nr_thp_failed); + count_vm_events(THP_MIGRATION_SPLIT, stats.nr_thp_split); + trace_mm_migrate_pages(stats.nr_succeeded, stats.nr_failed_pages, + stats.nr_thp_succeeded, stats.nr_thp_failed, + stats.nr_thp_split, mode, reason); if (ret_succeeded) - *ret_succeeded = nr_succeeded; + *ret_succeeded = stats.nr_succeeded; return rc; } From patchwork Tue Dec 27 00:28:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13081972 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16264C3DA79 for ; Tue, 27 Dec 2022 00:29:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 52B13940008; Mon, 26 Dec 2022 19:29:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 50A39940007; Mon, 26 Dec 2022 19:29:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3EFDD940008; Mon, 26 Dec 2022 19:29:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2D347940007 for ; Mon, 26 Dec 2022 19:29:34 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E6F191201F0 for ; Tue, 27 Dec 2022 00:29:33 +0000 (UTC) X-FDA: 80286202626.22.60DCDBF Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf09.hostedemail.com (Postfix) with ESMTP id 1534814000A for ; Tue, 27 Dec 2022 00:29:30 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=gxbO6+bv; spf=pass (imf09.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672100971; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EmUgQlD9tnG4P9yimIg6vRqU1RZS8A7fonhRfLBzs4c=; b=lKHdxsQmtj093IJlC5fWBiYa8BJws7SDJexnk8st4/+mqZ9U2LIu2j2JaUXwcl6HkVYrRt TbK+Sicpj0Z+4Cli2w9rSDhBoBnHeefm6eF0kuaKq0+KqIewDVlaU4yqpeOWU5ICAEo8pv 2v9x7ekWIeeqtNmfe00vcq0Fc5/6lHk= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=gxbO6+bv; spf=pass (imf09.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672100971; a=rsa-sha256; cv=none; b=H9XZccMH2lIG5Ui/ZxP0y8Oo3CE5pDv+5TNlN+u893Nfff7AsfE2clgzI4aBSkyFT21pCs KzZe9dYI2asaE7BOfkAl8GQ1nKRR8Fm2d9rPit7Ltj0evMF+q4LkNG4+zDsOphTTHbhGuj aUWkJqTuiUqLolfGKtLtk6cWt6sBWCQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672100971; x=1703636971; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fdSCkAQLw2skvmVvw4V6UUPknrIjo/5xEY/WxCdRv2U=; b=gxbO6+bvSuyUPfTcg0pZ9XtPVYqaJli/zJbqxttBTG2kwY5q+3Lezob0 /uTVs9QIriRzsZKL2/jEyriRWlQ/gyg4xu5DK0x/QnleqpjXpC+4ALwjJ kZH8BWjSN8ajs+wogEq8z3yCAeOCFGdNcxckglLVD4kkjA1x0WeHQwNBN 6BDQmHJZpK2COh7i2MLfezCc1XNw2sgV+Oa78sSn0fC3+YBTqCGLOzhVM KdF6VdfJ4jGmAGAH/2B2qYCMzPvdgtJ4dyQGKr1Jx1yrkcIL3Fq+Tx6/G jSW2/L1GQAN3Z0W5HVE0BafdpxzfvnQH4v84n1HuEKPWWuQeK2+2xZkIc g==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="322597216" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="322597216" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:29 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="760172194" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="760172194" Received: from yyang3-mobl1.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.212.104]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:25 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH 2/8] migrate_pages: separate hugetlb folios migration Date: Tue, 27 Dec 2022 08:28:53 +0800 Message-Id: <20221227002859.27740-3-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221227002859.27740-1-ying.huang@intel.com> References: <20221227002859.27740-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 1534814000A X-Stat-Signature: uudbkw4bhsngsrcixryuh814y7pbc548 X-HE-Tag: 1672100970-997455 X-HE-Meta: U2FsdGVkX1+SXy5aZ6oVduwykqWeet/nC4Jbnd98JayyTpyiMymJt/JD0cIkAroJHw+ulkwwi10wU0d5URPlvhGvoq0/3tEMaU1oAKe59K4dNP4Hh8lj47WiptEJvk4lF/V2hiveqR02rTEIV38G7YMjAN+b8Bc3z621ImI4NqpBoHSiuMNkGg/NLxWPCWN4l3IW4EB4rLf49F5oA9vKn67m82uO6M/p3Vj2WfIatYXV2Qy+R4v2lI9TMEz23soQ7927P13mfD+i4rqcNwRE0V4YsuFI1CvCIJzyfbX29+hnyeHnBHbRUQ4l7pfy/wAX2er0d+zqdPaKrB6rsZfvd9ULA07x/u4+frPZa6xl2DOxZjBkWfp/bXWrtDSfVt4szup0UYiv2yBu1lkLZk1xT+m45J90Dn/OkuGeClaJ41HnV+l/qfWhOqgfZsdVr4jV8GqwUrtYZ0e7PDCLaBdU5Z0pX4sjebDYP0DRyCurhkcLPaMQ+C8VP3Uy81mtGI+Zlzh9ENC7NttRGsSLRw0GLMIeUNaHQWCxy7jRqoXfs6/CAreXUUhI8YzFFa+qZkouAeYBfTxid949jM9jfHcyLnklu2sEIbEssxZz1XRa5NnL6LOpdbNZSjWBZO53fyOBMBdxYFsNUi+nQLjSvMHlTl1QBqtAeA1AarKuEuZykc88j9azVhFzWJnnP0BmeyjARTehdnAemy+vHzANoPabhPC9j+wIaAerodS3xLo7TcmXfeVsh/aPhkmhroOXyPDZ61y2vqzI2zsQ9Sm/Rp4IQzkAHcK5WDw+8zs9bQQYCldFVvCkIFYF49PaVvMQ1XCKY7xOt68JUq4OsAyf5S7H8FfY4bPw+yE9aeTdUyOZn9Ltlqt3/5g4fjYXUQbj5e2Ib/qUolwhNkQ+dIo+ACpaGF806WlcI7d1g3zg6WMqX64i6EoEEoLPpjHgqjjfxRRR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a preparation patch to batch the folio unmapping and moving for the non-hugetlb folios. Based on that we can batch the TLB shootdown during the folio migration and make it possible to use some hardware accelerator for the folio copying. In this patch the hugetlb folios and non-hugetlb folios migration is separated in migrate_pages() to make it easy to change the non-hugetlb folios migration implementation. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin --- mm/migrate.c | 114 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 99 insertions(+), 15 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index ec9263a33d38..bdbe73fe2eb7 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1404,6 +1404,87 @@ struct migrate_pages_stats { int nr_thp_split; }; +static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, + free_page_t put_new_page, unsigned long private, + enum migrate_mode mode, int reason, + struct migrate_pages_stats *stats, + struct list_head *ret_folios) +{ + int retry = 1; + int nr_failed = 0; + int nr_retry_pages = 0; + int pass = 0; + struct folio *folio, *folio2; + int rc = 0, nr_pages; + + for (pass = 0; pass < 10 && retry; pass++) { + retry = 0; + nr_retry_pages = 0; + + list_for_each_entry_safe(folio, folio2, from, lru) { + if (!folio_test_hugetlb(folio)) + continue; + + nr_pages = folio_nr_pages(folio); + + cond_resched(); + + rc = unmap_and_move_huge_page(get_new_page, + put_new_page, private, + &folio->page, pass > 2, mode, + reason, ret_folios); + /* + * The rules are: + * Success: hugetlb folio will be put back + * -EAGAIN: stay on the from list + * -ENOMEM: stay on the from list + * -ENOSYS: stay on the from list + * Other errno: put on ret_folios list + */ + switch(rc) { + case -ENOSYS: + /* Hugetlb migration is unsupported */ + nr_failed++; + stats->nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, ret_folios); + break; + case -ENOMEM: + /* + * When memory is low, don't bother to try to migrate + * other folios, just exit. + */ + nr_failed++; + stats->nr_failed_pages += nr_pages; + goto out; + case -EAGAIN: + retry++; + nr_retry_pages += nr_pages; + break; + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + break; + default: + /* + * Permanent failure (-EBUSY, etc.): + * unlike -EAGAIN case, the failed folio is + * removed from migration folio list and not + * retried in the next outer loop. + */ + nr_failed++; + stats->nr_failed_pages += nr_pages; + break; + } + } + } +out: + nr_failed += retry; + stats->nr_failed_pages += nr_retry_pages; + if (rc != -ENOMEM) + rc = nr_failed; + + return rc; +} + /* * migrate_pages - migrate the folios specified in a list, to the free folios * supplied as the target for the page migration @@ -1437,7 +1518,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, int retry = 1; int large_retry = 1; int thp_retry = 1; - int nr_failed = 0; + int nr_failed; int nr_retry_pages = 0; int nr_large_failed = 0; int pass = 0; @@ -1454,6 +1535,12 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, trace_mm_migrate_pages_start(mode, reason); memset(&stats, 0, sizeof(stats)); + rc = migrate_hugetlbs(from, get_new_page, put_new_page, private, mode, reason, + &stats, &ret_folios); + if (rc < 0) + goto out; + nr_failed = rc; + split_folio_migration: for (pass = 0; pass < 10 && (retry || large_retry); pass++) { retry = 0; @@ -1462,30 +1549,28 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_retry_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { + if (folio_test_hugetlb(folio)) { + list_move_tail(&folio->lru, &ret_folios); + continue; + } + /* * Large folio statistics is based on the source large * folio. Capture required information that might get * lost during migration. */ - is_large = folio_test_large(folio) && !folio_test_hugetlb(folio); + is_large = folio_test_large(folio); is_thp = is_large && folio_test_pmd_mappable(folio); nr_pages = folio_nr_pages(folio); + cond_resched(); - if (folio_test_hugetlb(folio)) - rc = unmap_and_move_huge_page(get_new_page, - put_new_page, private, - &folio->page, pass > 2, mode, - reason, - &ret_folios); - else - rc = unmap_and_move(get_new_page, put_new_page, - private, folio, pass > 2, mode, - reason, &ret_folios); + rc = unmap_and_move(get_new_page, put_new_page, + private, folio, pass > 2, mode, + reason, &ret_folios); /* * The rules are: - * Success: non hugetlb folio will be freed, hugetlb - * folio will be put back + * Success: folio will be freed * -EAGAIN: stay on the from list * -ENOMEM: stay on the from list * -ENOSYS: stay on the from list @@ -1512,7 +1597,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, stats.nr_thp_split += is_thp; break; } - /* Hugetlb migration is unsupported */ } else if (!no_split_folio_counting) { nr_failed++; } From patchwork Tue Dec 27 00:28:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13081973 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32B09C46467 for ; Tue, 27 Dec 2022 00:29:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C2E22940009; Mon, 26 Dec 2022 19:29:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BDED9940007; Mon, 26 Dec 2022 19:29:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A7F8A940009; Mon, 26 Dec 2022 19:29:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9BBFD940007 for ; Mon, 26 Dec 2022 19:29:37 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 7274F40412 for ; Tue, 27 Dec 2022 00:29:37 +0000 (UTC) X-FDA: 80286202794.12.A7C0CB8 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf19.hostedemail.com (Postfix) with ESMTP id 80BB41A0005 for ; Tue, 27 Dec 2022 00:29:35 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=fSVtz34Y; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672100975; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i7Wl0sDq2yeTgMud/WS7D/Daca741nvRe9z60L9ZlCY=; b=dcLnxSfdkXq0iI1Lu5R8b4Yf2HWZ4teEbIHfPR9NMelk9ue26Vxcbv4ZRkraTm7TiwVrA7 YclYWdHZmTpNJ1QyupC+1/XSuCtyvmpetsy46d1QBn906O/PJvEj9YSo7PZ4J0BXTqjoC1 nzyZ/VbFsatsm1iCBu6zCaW2bEw/Vc8= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=fSVtz34Y; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672100975; a=rsa-sha256; cv=none; b=gg3usnWpb5YEueN05Iik/YYVUqWOVAkZGmTwejKTxBpBCh7uIIYMdSGbS04HTIj3xeU6B5 Nb5ko8OUJAj9KHCxLLCELRbdFRiQYGJ8mpMYShajh3dMYjNHaOdakOHW8xWuMrx0IXbn3Q FaZbPT9AqEC2C4IRKi7ha4SKcg3ZRSw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672100975; x=1703636975; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=cdwO6h42qElO4zPv01NpyCEn7VoA3Yf9bDuT74L7CAA=; b=fSVtz34YaLrh27PNnrwYQk5gUkVpldDcnTC2qb29BgOLALre1bC4+ui8 IkuhGFfFWT9g2seuuzohIuT8GHpcuDIvOogSfeJQzTsNyRARNczbYnG/P FcXG3LZo/0i2preeZWO8/AJxaX/7rf8D8G39dW+4DlTdvd/LgyTbGpj0/ Bl5w1PGixbB5L/PdryV+T9HI7xf9MMjQDk+z7Z7jIgeTbTDmDxNdsG3YZ pe4C3q+w50qPVD4xpbti14i3XA7EOAIKE/Ui5UxThQLtZqdVbaAhMUzVa PPTsjggwpkn29YEiOSJ4lTJOKOq0YD97QEY1LlUWTxVOUA1UUHIMQPT+a A==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="322597231" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="322597231" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:34 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="760172204" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="760172204" Received: from yyang3-mobl1.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.212.104]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:30 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH 3/8] migrate_pages: restrict number of pages to migrate in batch Date: Tue, 27 Dec 2022 08:28:54 +0800 Message-Id: <20221227002859.27740-4-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221227002859.27740-1-ying.huang@intel.com> References: <20221227002859.27740-1-ying.huang@intel.com> MIME-Version: 1.0 X-Stat-Signature: dhz1kpshwbwqmiepaho7o8x97xau9m36 X-Rspam-User: X-Rspamd-Queue-Id: 80BB41A0005 X-Rspamd-Server: rspam06 X-HE-Tag: 1672100975-380888 X-HE-Meta: U2FsdGVkX18CaE1uPBNA/ckE5GxC0aCxYExE+Iaxfbvz/UgvgUtSelWTrAc/bUIyqYJ46TzBLqMSb2zdBbE4IPNsKYe2uOYZtxeaGFsiqMa7fWkvviBDAj0onn+X5gljvopLG8xNrXgFdmbeqwu1nYNoJOpEZmIzCLPelIKBt7bFtjcvMJW0XYth0BxcmBoH4umtgvZ7AySpYtsAptB4loxk9ILBmoPXGEqyXT4DlBG5dhXy4HBzOjcjMjo+5f5ERZabuE21y9epkQRBXoR6FWkB2jWLOuQ01bQzeFwayB6p/U5q5qSoEsE9ulTxBWH7ZKY83ztjUNVxwzJZTV2Ubq0eo4GjzoSagNGH3Jf5er47cxgEkj3AkL07zbpt08h+vAfy/YD4Vq5cMZ9aMuzqLHpH4Uag7BOqSG8sO4uftVVjHBBNaUJtKAdVvumORXH+Zvlz484PU+fOCnFF3PpUJrN1nZyq+25YPfctx0vI1ylJR0JPgK5f29ZclfK0Oz1kJmKsxEQjqFNnFkHFHCP1V1YTJXLKNHfSvaO6Kqfe3Kc2at1AicOZoxlqzKLJgaL4WiwPyyeZ2J01q90hilQxRgEldQJwO1iXcddth6bg+lvRV5DwxoMjueAfsyk6mFzvVOv7RogMg7Erd0BRhbgjX5u2zBk6ieMhzf5bb/fabg6Rz8qkImhHp2Lfu9HypsKtom7A/DlexDVaD9McR2RrFy1TZPZW8V3U/iv3vMtfb16Vfgrfslv9Es9EHpjY9WgB4lsRm8ZZ6znacBks1hBhXHX1uQsPb6MnsklPd1ei3J3/30RsHNLDYlFyqCKyezeBLEc1hiEqNgrQFHQeWtsR4nK2uaPWcIPOkuCfluzSSBtntFZp14mB7xdznE5UV78ZW9FRmJrH1hpZN9hvJQIKopoCRx5hAeswHd3ibfdpynzyqQEDlQvSnoNK6bCI7icEIHQfWXmMoQo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a preparation patch to batch the folio unmapping and moving for non-hugetlb folios. If we had batched the folio unmapping, all folios to be migrated would be unmapped before copying the contents and flags of the folios. If the folios that were passed to migrate_pages() were too many in unit of pages, the execution of the processes would be stopped for too long time, thus too long latency. For example, migrate_pages() syscall will call migrate_pages() with all folios of a process. To avoid this possible issue, in this patch, we restrict the number of pages to be migrated to be no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin --- mm/migrate.c | 173 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 106 insertions(+), 67 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index bdbe73fe2eb7..97ea0737ab2b 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1485,40 +1485,15 @@ static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, return rc; } -/* - * migrate_pages - migrate the folios specified in a list, to the free folios - * supplied as the target for the page migration - * - * @from: The list of folios to be migrated. - * @get_new_page: The function used to allocate free folios to be used - * as the target of the folio migration. - * @put_new_page: The function used to free target folios if migration - * fails, or NULL if no special handling is necessary. - * @private: Private data to be passed on to get_new_page() - * @mode: The migration mode that specifies the constraints for - * folio migration, if any. - * @reason: The reason for folio migration. - * @ret_succeeded: Set to the number of folios migrated successfully if - * the caller passes a non-NULL pointer. - * - * The function returns after 10 attempts or if no folios are movable any more - * because the list has become empty or no retryable folios exist any more. - * It is caller's responsibility to call putback_movable_pages() to return folios - * to the LRU or free list only if ret != 0. - * - * Returns the number of {normal folio, large folio, hugetlb} that were not - * migrated, or an error code. The number of large folio splits will be - * considered as the number of non-migrated large folio, no matter how many - * split folios of the large folio are migrated successfully. - */ -int migrate_pages(struct list_head *from, new_page_t get_new_page, +static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, free_page_t put_new_page, unsigned long private, - enum migrate_mode mode, int reason, unsigned int *ret_succeeded) + enum migrate_mode mode, int reason, struct list_head *ret_folios, + struct migrate_pages_stats *stats) { int retry = 1; int large_retry = 1; int thp_retry = 1; - int nr_failed; + int nr_failed = 0; int nr_retry_pages = 0; int nr_large_failed = 0; int pass = 0; @@ -1526,20 +1501,9 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, bool is_thp = false; struct folio *folio, *folio2; int rc, nr_pages; - LIST_HEAD(ret_folios); LIST_HEAD(split_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); bool no_split_folio_counting = false; - struct migrate_pages_stats stats; - - trace_mm_migrate_pages_start(mode, reason); - - memset(&stats, 0, sizeof(stats)); - rc = migrate_hugetlbs(from, get_new_page, put_new_page, private, mode, reason, - &stats, &ret_folios); - if (rc < 0) - goto out; - nr_failed = rc; split_folio_migration: for (pass = 0; pass < 10 && (retry || large_retry); pass++) { @@ -1549,11 +1513,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_retry_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { - if (folio_test_hugetlb(folio)) { - list_move_tail(&folio->lru, &ret_folios); - continue; - } - /* * Large folio statistics is based on the source large * folio. Capture required information that might get @@ -1567,15 +1526,14 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, rc = unmap_and_move(get_new_page, put_new_page, private, folio, pass > 2, mode, - reason, &ret_folios); + reason, ret_folios); /* * The rules are: * Success: folio will be freed * -EAGAIN: stay on the from list * -ENOMEM: stay on the from list * -ENOSYS: stay on the from list - * Other errno: put on ret_folios list then splice to - * from list + * Other errno: put on ret_folios list */ switch(rc) { /* @@ -1592,17 +1550,17 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, /* Large folio migration is unsupported */ if (is_large) { nr_large_failed++; - stats.nr_thp_failed += is_thp; + stats->nr_thp_failed += is_thp; if (!try_split_folio(folio, &split_folios)) { - stats.nr_thp_split += is_thp; + stats->nr_thp_split += is_thp; break; } } else if (!no_split_folio_counting) { nr_failed++; } - stats.nr_failed_pages += nr_pages; - list_move_tail(&folio->lru, &ret_folios); + stats->nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, ret_folios); break; case -ENOMEM: /* @@ -1611,13 +1569,13 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (is_large) { nr_large_failed++; - stats.nr_thp_failed += is_thp; + stats->nr_thp_failed += is_thp; /* Large folio NUMA faulting doesn't split to retry. */ if (!nosplit) { int ret = try_split_folio(folio, &split_folios); if (!ret) { - stats.nr_thp_split += is_thp; + stats->nr_thp_split += is_thp; break; } else if (reason == MR_LONGTERM_PIN && ret == -EAGAIN) { @@ -1635,17 +1593,17 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_failed++; } - stats.nr_failed_pages += nr_pages + nr_retry_pages; + stats->nr_failed_pages += nr_pages + nr_retry_pages; /* * There might be some split folios of fail-to-migrate large - * folios left in split_folios list. Move them back to migration + * folios left in split_folios list. Move them to ret_folios * list so that they could be put back to the right list by * the caller otherwise the folio refcnt will be leaked. */ - list_splice_init(&split_folios, from); + list_splice_init(&split_folios, ret_folios); /* nr_failed isn't updated for not used */ nr_large_failed += large_retry; - stats.nr_thp_failed += thp_retry; + stats->nr_thp_failed += thp_retry; goto out; case -EAGAIN: if (is_large) { @@ -1657,8 +1615,8 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_retry_pages += nr_pages; break; case MIGRATEPAGE_SUCCESS: - stats.nr_succeeded += nr_pages; - stats.nr_thp_succeeded += is_thp; + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; break; default: /* @@ -1669,20 +1627,20 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (is_large) { nr_large_failed++; - stats.nr_thp_failed += is_thp; + stats->nr_thp_failed += is_thp; } else if (!no_split_folio_counting) { nr_failed++; } - stats.nr_failed_pages += nr_pages; + stats->nr_failed_pages += nr_pages; break; } } } nr_failed += retry; nr_large_failed += large_retry; - stats.nr_thp_failed += thp_retry; - stats.nr_failed_pages += nr_retry_pages; + stats->nr_thp_failed += thp_retry; + stats->nr_failed_pages += nr_retry_pages; /* * Try to migrate split folios of fail-to-migrate large folios, no * nr_failed counting in this round, since all split folios of a @@ -1693,7 +1651,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, * Move non-migrated folios (after 10 retries) to ret_folios * to avoid migrating them again. */ - list_splice_init(from, &ret_folios); + list_splice_init(from, ret_folios); list_splice_init(&split_folios, from); no_split_folio_counting = true; retry = 1; @@ -1701,6 +1659,87 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, } rc = nr_failed + nr_large_failed; +out: + return rc; +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#define NR_MAX_BATCHED_MIGRATION HPAGE_PMD_NR +#else +#define NR_MAX_BATCHED_MIGRATION 512 +#endif + +/* + * migrate_pages - migrate the folios specified in a list, to the free folios + * supplied as the target for the page migration + * + * @from: The list of folios to be migrated. + * @get_new_page: The function used to allocate free folios to be used + * as the target of the folio migration. + * @put_new_page: The function used to free target folios if migration + * fails, or NULL if no special handling is necessary. + * @private: Private data to be passed on to get_new_page() + * @mode: The migration mode that specifies the constraints for + * folio migration, if any. + * @reason: The reason for folio migration. + * @ret_succeeded: Set to the number of folios migrated successfully if + * the caller passes a non-NULL pointer. + * + * The function returns after 10 attempts or if no folios are movable any more + * because the list has become empty or no retryable folios exist any more. + * It is caller's responsibility to call putback_movable_pages() to return folios + * to the LRU or free list only if ret != 0. + * + * Returns the number of {normal folio, large folio, hugetlb} that were not + * migrated, or an error code. The number of large folio splits will be + * considered as the number of non-migrated large folio, no matter how many + * split folios of the large folio are migrated successfully. + */ +int migrate_pages(struct list_head *from, new_page_t get_new_page, + free_page_t put_new_page, unsigned long private, + enum migrate_mode mode, int reason, unsigned int *ret_succeeded) +{ + int rc, rc_gether; + int nr_pages; + struct folio *folio, *folio2; + LIST_HEAD(folios); + LIST_HEAD(ret_folios); + struct migrate_pages_stats stats; + + trace_mm_migrate_pages_start(mode, reason); + + memset(&stats, 0, sizeof(stats)); + + rc_gether = migrate_hugetlbs(from, get_new_page, put_new_page, private, + mode, reason, &stats, &ret_folios); + if (rc_gether < 0) + goto out; +again: + nr_pages = 0; + list_for_each_entry_safe(folio, folio2, from, lru) { + if (folio_test_hugetlb(folio)) { + list_move_tail(&folio->lru, &ret_folios); + continue; + } + + nr_pages += folio_nr_pages(folio); + if (nr_pages > NR_MAX_BATCHED_MIGRATION) + break; + } + if (nr_pages > NR_MAX_BATCHED_MIGRATION) + list_cut_before(&folios, from, &folio->lru); + else + list_splice_init(from, &folios); + rc = migrate_pages_batch(&folios, get_new_page, put_new_page, private, + mode, reason, &ret_folios, &stats); + list_splice_tail_init(&folios, &ret_folios); + if (rc < 0) { + rc_gether = rc; + goto out; + } + rc_gether += rc; + if (!list_empty(from)) + goto again; out: /* * Put the permanent failure folio back to migration list, they @@ -1713,7 +1752,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, * are migrated successfully. */ if (list_empty(from)) - rc = 0; + rc_gether = 0; count_vm_events(PGMIGRATE_SUCCESS, stats.nr_succeeded); count_vm_events(PGMIGRATE_FAIL, stats.nr_failed_pages); @@ -1727,7 +1766,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, if (ret_succeeded) *ret_succeeded = stats.nr_succeeded; - return rc; + return rc_gether; } struct page *alloc_migration_target(struct page *page, unsigned long private) From patchwork Tue Dec 27 00:28:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13081974 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9E3EC4332F for ; Tue, 27 Dec 2022 00:29:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 37B7394000A; Mon, 26 Dec 2022 19:29:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 32C0C940007; Mon, 26 Dec 2022 19:29:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1CD4594000A; Mon, 26 Dec 2022 19:29:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0FCC5940007 for ; Mon, 26 Dec 2022 19:29:41 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id CCAE8803C1 for ; Tue, 27 Dec 2022 00:29:40 +0000 (UTC) X-FDA: 80286202920.20.A812066 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf19.hostedemail.com (Postfix) with ESMTP id D5F7F1A0004 for ; Tue, 27 Dec 2022 00:29:38 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=MfIFT1GN; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672100979; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ObvAcUhsiB4wXEoUkidIw7PHh73SCgJmibqHOUeVrgY=; b=wmc/c0JlxrpDufEIkgqDUZ6UXNX2+yYhxvO2rzTfaxwLIkyL3/QeFTADLiY14LjHMSRlLC uTU/T89OYCWpY49gpYS2tQDL635c56+WvAdeQHFKGyRZYoCY47c8HeOpmVbjMQR6e0YOCS Wk7IdzITdrQPVi2kBpCZ5VM7EpiAgtc= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=MfIFT1GN; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672100979; a=rsa-sha256; cv=none; b=OkZh1oNjDmAd/LoGL1yYhGHz99XjtQFmR6w7C724/puA/c7hBhanywXHmXvcMxQvsIFv5m Obx6klUaTHnh/5A9gyiiI1BpigviVAekSTJhSKQhuZl8VR+4oqvcZxU9R/xxZJNiRchR8K d8jiP8j8N2ho7Mv+mCYhU/VxD52j5DQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672100978; x=1703636978; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=IlUDC48YZthy4iB9f936bSNt5z2BK2NSyqzBAjVYcv8=; b=MfIFT1GN/KfQX/KXvbAGSkrSnO4NUFE9QRR8lgsUE/UD35lRDVxZdpJT 4+A1ZgYm9wZiY5Sh4Ov42G4RSZyW6ARx9O9JdkzCIzEStyvPgAicqplEQ dEYwhqB3GNhIauTujU9R5bY5zwNf6KBegzLyGz1LQ52scWpfh0pTa1Oez gEkzHUrBPlQYn95jXRM+l56Ij2wJayOI+4g+kxXs3H0F42JbJNEknOmb1 1GgFTjVrVb6+VDr5NtKHz5vrIeEB6PwscOFT3TotjX5udnI7Fj5UGA/bZ MJIG53aTgbni/8otsZfCm4Gkol6/JTwR+d6sf+10cKGJ3JU/8uptNUtlB A==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="322597243" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="322597243" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:38 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="760172216" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="760172216" Received: from yyang3-mobl1.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.212.104]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:34 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH 4/8] migrate_pages: split unmap_and_move() to _unmap() and _move() Date: Tue, 27 Dec 2022 08:28:55 +0800 Message-Id: <20221227002859.27740-5-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221227002859.27740-1-ying.huang@intel.com> References: <20221227002859.27740-1-ying.huang@intel.com> MIME-Version: 1.0 X-Stat-Signature: w8krntbiqr31w4b3t1aq5ubpie3mhzus X-Rspam-User: X-Rspamd-Queue-Id: D5F7F1A0004 X-Rspamd-Server: rspam06 X-HE-Tag: 1672100978-709092 X-HE-Meta: U2FsdGVkX1/G78P0XwzYF3HwrWtdtBhc6aDSgOj5C8Yb8inBADv14IqrjCgHZtVEg62qBeMu8ulX/Z/CNpZU+/y6FzSuhXXmuJB2r1l+LlEJEJ3G/p7JwW6h/SVeNdusnR6k47rzgtFIburbPErzWbO+1HlqWbUQX30vBMozXRYLbnuyeyO1StJRRauSqeqtpfKUCSxeAzQfhd2JuQKSJmXwW8MVFjjbP9D3wWXNQk0PSKeSkWkL1pjezoTbv38Jxda2QtStMnMgumDaju9cyNxnANPfaTxef5RUgAtlD0ADezXFAr3/w7/RYA0OK6yfSkuofGQEG+uQw6VEhWaQWsKp+3NN8Gjy87ZbbU+1xNb3VwqzIgFZ9c+Bj5HRLh4FoB/dGiEl5mROm0M/g9YeVoY+/4d8xOk6U3tjniVS/kNGWcewYV/P1Ky7rRu7a5N96w5TsF/WJaq4a1sXC6bB7g+1juDOh63iQ9cUM/duu0DQKam10cClERJP0GUAVhs6cfQebz1EEBZmKrC/yic8gXnqlFmmQ0YhfuEHir1+gYcw5bMvPvbZ9CHNVWNtNqQWPv8M5yAfgmeIOOJQIOgAkt85GJ/FSbVR5+DkMGVopKlZhGRYRplidyfKEawIu6f0Ko9nBc1Tac05xbkyoooOwfjxDKmNyTwnaedWOdjpsYlKiwhE3ZdWy81YjaOwPJyJq1lDHLrpL6ski0err+MEptl16RLV2YITaf34C+awn/PlDrN/61jEU6O27psgBzCSyj4zww/PeAhqnMFT4wR9yQXEgYHS0W2tJ3Z7o92CVTzp3LkT83JhG3bJIdAhSRm3+t2pXHv6uUoVKMXptDi+9BI55lDU1a8gTqsVNjx9Ap478xWZO3cUzh6abhOLaN4CW27hK2D7q37F5nQKR78mikry5l2EJKSnHl0mxJrBklMydfIUfCj0rG7ANM1hHTIpXdcwoq2Exkk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a preparation patch to batch the folio unmapping and moving. In this patch, unmap_and_move() is split to migrate_folio_unmap() and migrate_folio_move(). So, we can batch _unmap() and _move() in different loops later. To pass some information between unmap and move, the original unused dst->mapping and dst->private are used. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin --- include/linux/migrate.h | 1 + mm/migrate.c | 162 +++++++++++++++++++++++++++++----------- 2 files changed, 121 insertions(+), 42 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 3ef77f52a4f0..7376074f2e1e 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -18,6 +18,7 @@ struct migration_target_control; * - zero on page migration success; */ #define MIGRATEPAGE_SUCCESS 0 +#define MIGRATEPAGE_UNMAP 1 /** * struct movable_operations - Driver page migration diff --git a/mm/migrate.c b/mm/migrate.c index 97ea0737ab2b..e2383b430932 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1009,11 +1009,29 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, return rc; } -static int __unmap_and_move(struct folio *src, struct folio *dst, +static void __migrate_folio_record(struct folio *dst, + unsigned long page_was_mapped, + struct anon_vma *anon_vma) +{ + dst->mapping = (struct address_space *)anon_vma; + dst->private = (void *)page_was_mapped; +} + +static void __migrate_folio_extract(struct folio *dst, + int *page_was_mappedp, + struct anon_vma **anon_vmap) +{ + *anon_vmap = (struct anon_vma *)dst->mapping; + *page_was_mappedp = (unsigned long)dst->private; + dst->mapping = NULL; + dst->private = NULL; +} + +static int __migrate_folio_unmap(struct folio *src, struct folio *dst, int force, enum migrate_mode mode) { int rc = -EAGAIN; - bool page_was_mapped = false; + int page_was_mapped = 0; struct anon_vma *anon_vma = NULL; bool is_lru = !__PageMovable(&src->page); @@ -1089,8 +1107,8 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, goto out_unlock; if (unlikely(!is_lru)) { - rc = move_to_new_folio(dst, src, mode); - goto out_unlock_both; + __migrate_folio_record(dst, page_was_mapped, anon_vma); + return MIGRATEPAGE_UNMAP; } /* @@ -1115,11 +1133,40 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, VM_BUG_ON_FOLIO(folio_test_anon(src) && !folio_test_ksm(src) && !anon_vma, src); try_to_migrate(src, 0); - page_was_mapped = true; + page_was_mapped = 1; } - if (!folio_mapped(src)) - rc = move_to_new_folio(dst, src, mode); + if (!folio_mapped(src)) { + __migrate_folio_record(dst, page_was_mapped, anon_vma); + return MIGRATEPAGE_UNMAP; + } + + + if (page_was_mapped) + remove_migration_ptes(src, src, false); + +out_unlock_both: + folio_unlock(dst); +out_unlock: + /* Drop an anon_vma reference if we took one */ + if (anon_vma) + put_anon_vma(anon_vma); + folio_unlock(src); +out: + + return rc; +} + +static int __migrate_folio_move(struct folio *src, struct folio *dst, + enum migrate_mode mode) +{ + int rc; + int page_was_mapped = 0; + struct anon_vma *anon_vma = NULL; + + __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); + + rc = move_to_new_folio(dst, src, mode); /* * When successful, push dst to LRU immediately: so that if it @@ -1140,14 +1187,11 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, remove_migration_ptes(src, rc == MIGRATEPAGE_SUCCESS ? dst : src, false); -out_unlock_both: folio_unlock(dst); -out_unlock: /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); folio_unlock(src); -out: /* * If migration is successful, decrease refcount of dst, * which will not free the page because new page owner increased @@ -1159,19 +1203,32 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, return rc; } -/* - * Obtain the lock on folio, remove all ptes and migrate the folio - * to the newly allocated folio in dst. - */ -static int unmap_and_move(new_page_t get_new_page, - free_page_t put_new_page, - unsigned long private, struct folio *src, - int force, enum migrate_mode mode, - enum migrate_reason reason, - struct list_head *ret) +static void migrate_folio_done(struct folio *src, + enum migrate_reason reason) +{ + /* + * Compaction can migrate also non-LRU pages which are + * not accounted to NR_ISOLATED_*. They can be recognized + * as __PageMovable + */ + if (likely(!__folio_test_movable(src))) + mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON + + folio_is_file_lru(src), -folio_nr_pages(src)); + + if (reason != MR_MEMORY_FAILURE) + /* We release the page in page_handle_poison. */ + folio_put(src); +} + +/* Obtain the lock on page, remove all ptes. */ +static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, + unsigned long private, struct folio *src, + struct folio **dstp, int force, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) { struct folio *dst; - int rc = MIGRATEPAGE_SUCCESS; + int rc = MIGRATEPAGE_UNMAP; struct page *newpage = NULL; if (!thp_migration_supported() && folio_test_transhuge(src)) @@ -1182,20 +1239,50 @@ static int unmap_and_move(new_page_t get_new_page, folio_clear_active(src); folio_clear_unevictable(src); /* free_pages_prepare() will clear PG_isolated. */ - goto out; + list_del(&src->lru); + migrate_folio_done(src, reason); + return MIGRATEPAGE_SUCCESS; } newpage = get_new_page(&src->page, private); if (!newpage) return -ENOMEM; dst = page_folio(newpage); + *dstp = dst; dst->private = NULL; - rc = __unmap_and_move(src, dst, force, mode); + rc = __migrate_folio_unmap(src, dst, force, mode); + if (rc == MIGRATEPAGE_UNMAP) + return rc; + + /* + * A page that has not been migrated will have kept its + * references and be restored. + */ + /* restore the folio to right list. */ + if (rc != -EAGAIN) + list_move_tail(&src->lru, ret); + + if (put_new_page) + put_new_page(&dst->page, private); + else + folio_put(dst); + + return rc; +} + +/* Migrate the folio to the newly allocated folio in dst. */ +static int migrate_folio_move(free_page_t put_new_page, unsigned long private, + struct folio *src, struct folio *dst, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) +{ + int rc; + + rc = __migrate_folio_move(src, dst, mode); if (rc == MIGRATEPAGE_SUCCESS) set_page_owner_migrate_reason(&dst->page, reason); -out: if (rc != -EAGAIN) { /* * A folio that has been migrated has all references @@ -1211,20 +1298,7 @@ static int unmap_and_move(new_page_t get_new_page, * we want to retry. */ if (rc == MIGRATEPAGE_SUCCESS) { - /* - * Compaction can migrate also non-LRU folios which are - * not accounted to NR_ISOLATED_*. They can be recognized - * as __folio_test_movable - */ - if (likely(!__folio_test_movable(src))) - mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON + - folio_is_file_lru(src), -folio_nr_pages(src)); - - if (reason != MR_MEMORY_FAILURE) - /* - * We release the folio in page_handle_poison. - */ - folio_put(src); + migrate_folio_done(src, reason); } else { if (rc != -EAGAIN) list_add_tail(&src->lru, ret); @@ -1499,7 +1573,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, int pass = 0; bool is_large = false; bool is_thp = false; - struct folio *folio, *folio2; + struct folio *folio, *folio2, *dst = NULL; int rc, nr_pages; LIST_HEAD(split_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); @@ -1524,9 +1598,13 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, cond_resched(); - rc = unmap_and_move(get_new_page, put_new_page, - private, folio, pass > 2, mode, - reason, ret_folios); + rc = migrate_folio_unmap(get_new_page, put_new_page, private, + folio, &dst, pass > 2, mode, + reason, ret_folios); + if (rc == MIGRATEPAGE_UNMAP) + rc = migrate_folio_move(put_new_page, private, + folio, dst, mode, + reason, ret_folios); /* * The rules are: * Success: folio will be freed From patchwork Tue Dec 27 00:28:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13081975 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C6A7C4332F for ; Tue, 27 Dec 2022 00:29:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 96F1494000B; Mon, 26 Dec 2022 19:29:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 91F0C940007; Mon, 26 Dec 2022 19:29:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C08594000B; Mon, 26 Dec 2022 19:29:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6B8C8940007 for ; Mon, 26 Dec 2022 19:29:45 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3F4D6A07EE for ; Tue, 27 Dec 2022 00:29:45 +0000 (UTC) X-FDA: 80286203130.07.91E0988 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf19.hostedemail.com (Postfix) with ESMTP id 4277A1A0004 for ; Tue, 27 Dec 2022 00:29:43 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=b2EW0CfY; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672100983; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0G8H7d0G9UNdlos7DlSHKt5nvX7Cx0w//InGrHZNo3A=; b=usRTHJpAmW28Q/jC1KUBaaMpLXoZZcdoXYYgSDwzgdSC/wPZJBICxLjNR7Dzzba5s7Dyx1 8bsEuj+HX4jLIlofdYxWbfUhG450rBr9HzO/sWAwyeLnxRKQ4bqUv0A3BkdBOmSGXKfhi0 uCowE9XPcpUioGFH9iqne/yOgqYyUj0= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=b2EW0CfY; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672100983; a=rsa-sha256; cv=none; b=kb0IBINGh3/LrZvBPWRZEv7ZBnHgbpE+pVVjvMVWA3Gn22IpSAWTD1T+JL5lYtKxI7MvV3 6jpkrhb8mlo8HotAoOsT8Gsf6MELyTkOLsd+Dsk7CQ57AcLD5P69e5V2dYXN4RnwVEU0Mg HyShmCmg2D0ESekMel/yBH4A2OMPOoE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672100983; x=1703636983; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=KtvZc7fLcCQg6k7ytQ3p66ghNvglTTCVegaLh+l+zaA=; b=b2EW0CfYHElMyq0lUKI6FhwG08hLfKh50NZL5Tqx8DQTqXNTz/csF4Ys KeWnC1QccI1npo9Bu3bVvn8IYgoZrdCJkTgmpVSC2Do0Oay0IZB6t5SrA 90ii3FWaA8Eyp74aMWi/4ZxlxEl4s7j9FbvwSfhrwNS1QZ7bTEWQQ4a3k aG4j0JSmuAQCDTmu+eZVdA0L9E0CXa/g4bcb7sphmDSfJ2anp8VIWTBfl BqJ2iM9bm3nhvdBmsziBgL4WH+WC09G401tjk5p6vm6RL1zPhhPhm9v5K +juieLUIOo5HJWTgDam7ohhMQUKFmqtinDcetHQYIoWV2MikU8NP3hTlu w==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="322597255" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="322597255" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:42 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="760172228" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="760172228" Received: from yyang3-mobl1.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.212.104]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:38 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH 5/8] migrate_pages: batch _unmap and _move Date: Tue, 27 Dec 2022 08:28:56 +0800 Message-Id: <20221227002859.27740-6-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221227002859.27740-1-ying.huang@intel.com> References: <20221227002859.27740-1-ying.huang@intel.com> MIME-Version: 1.0 X-Stat-Signature: ee7hw6qxum4h4txrizxbo65cxdk9kz6b X-Rspam-User: X-Rspamd-Queue-Id: 4277A1A0004 X-Rspamd-Server: rspam06 X-HE-Tag: 1672100983-838852 X-HE-Meta: U2FsdGVkX18i09uV52ufb0tboiRXNcMiwFDuscFn3rjUtAWZsM0fp9pCpRarPmrPHUBJUhh0kdHNj/kefgNWlxTWcd99+cgK4MEncvMjSGcMJaIxyiiGtAXTiBhAOdbuQQKcn46jQwvBisbP6stFOKmCpuHXkuX1kBViGJQqd8rFgxA6X1tgqBnbtNmIwlaEDSHlOI2EgJbDJZ+N84SjJ93F51rz4IZlNiCMUCC1zMMDKv8qob5/i3shP4xGYse91uAtZKA71aOuWvqXuCpQZOk5Vwu1ozy0ZFIC0S9RHFj4z15jN1cRf3hy/L+8t0Qn+FfbX5DEDoVKDVZR4XMS7jG+b9GHX9Jjz0cvS0hqQtQsa0ehMpXhHu2aPNS3wta5qA9L2yC6VKzNnQF8aXt33Xtm54DHIvnj5uT45WktdhKZYz42dz3EzH4nHi1vVR0TY3KKDb0uZuG2sBnJ+k8DHWuTCbBAJKAbYPrfEY52ZuzFSpaPQyiRetcfn5FfWX25JM1MvqI0+R1NdLLNqt7dtVZ8ysIfAAICYUtJ0mCKwd/jLfxSzTLPX+Of1Q3FJCsXtF6GqhmMQztkQqWPhf28yROTrM+iBHK/dI9xQRBd19eMR+AS/s8d0HILep8DywqsCCEwtdZSMJAIBSW3tUFNVHL/teL2m+zlrVOuqvvm9AXrC5WBjbMGV2VF0rYO5pfVoLx1k0NbmDACAC57etBmMc85bLApAUHb324yG3RxSwo/14qeLsSYV6CUzvwyBh6yV1g7ymsgYiGtJ85VMfb4SDfjYrZQFAo+IslrRuF1MFwdCM7gBCopHASI+FgdiddG9KP93+lIkZrWY9UqRIb2n6iKZepS0EKWu3xrTm6j/RPyDweVY8c2g8fcO0NqBKAZAV/fWbRLdUZq7BrihWrR8oss1EVQnDnWRMkoc9camW+Idertc2Q5gO48+uqsd4xTfhnDi0weCLk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In this patch the _unmap and _move stage of the folio migration is batched. That for, previously, it is, for each folio _unmap() _move() Now, it is, for each folio _unmap() for each folio _move() Based on this, we can batch the TLB flushing and use some hardware accelerator to copy folios between batched _unmap and batched _move stages. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin --- mm/migrate.c | 189 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 165 insertions(+), 24 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index e2383b430932..dd68c3de3da8 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1027,8 +1027,32 @@ static void __migrate_folio_extract(struct folio *dst, dst->private = NULL; } +static void migrate_folio_undo_src(struct folio *src, + int page_was_mapped, + struct anon_vma *anon_vma, + struct list_head *ret) +{ + if (page_was_mapped) + remove_migration_ptes(src, src, false); + if (anon_vma) + put_anon_vma(anon_vma); + folio_unlock(src); + list_move_tail(&src->lru, ret); +} + +static void migrate_folio_undo_dst(struct folio *dst, + free_page_t put_new_page, + unsigned long private) +{ + folio_unlock(dst); + if (put_new_page) + put_new_page(&dst->page, private); + else + folio_put(dst); +} + static int __migrate_folio_unmap(struct folio *src, struct folio *dst, - int force, enum migrate_mode mode) + int force, bool force_lock, enum migrate_mode mode) { int rc = -EAGAIN; int page_was_mapped = 0; @@ -1055,6 +1079,11 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, if (current->flags & PF_MEMALLOC) goto out; + if (!force_lock) { + rc = -EDEADLOCK; + goto out; + } + folio_lock(src); } @@ -1168,6 +1197,8 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, rc = move_to_new_folio(dst, src, mode); + if (rc != -EAGAIN) + list_del(&dst->lru); /* * When successful, push dst to LRU immediately: so that if it * turns out to be an mlocked page, remove_migration_ptes() will @@ -1183,6 +1214,11 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, lru_add_drain(); } + if (rc == -EAGAIN) { + __migrate_folio_record(dst, page_was_mapped, anon_vma); + return rc; + } + if (page_was_mapped) remove_migration_ptes(src, rc == MIGRATEPAGE_SUCCESS ? dst : src, false); @@ -1223,7 +1259,7 @@ static void migrate_folio_done(struct folio *src, /* Obtain the lock on page, remove all ptes. */ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, unsigned long private, struct folio *src, - struct folio **dstp, int force, + struct folio **dstp, int force, bool force_lock, enum migrate_mode mode, enum migrate_reason reason, struct list_head *ret) { @@ -1251,7 +1287,7 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page *dstp = dst; dst->private = NULL; - rc = __migrate_folio_unmap(src, dst, force, mode); + rc = __migrate_folio_unmap(src, dst, force, force_lock, mode); if (rc == MIGRATEPAGE_UNMAP) return rc; @@ -1260,7 +1296,7 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page * references and be restored. */ /* restore the folio to right list. */ - if (rc != -EAGAIN) + if (rc != -EAGAIN && rc != -EDEADLOCK) list_move_tail(&src->lru, ret); if (put_new_page) @@ -1299,9 +1335,8 @@ static int migrate_folio_move(free_page_t put_new_page, unsigned long private, */ if (rc == MIGRATEPAGE_SUCCESS) { migrate_folio_done(src, reason); - } else { - if (rc != -EAGAIN) - list_add_tail(&src->lru, ret); + } else if (rc != -EAGAIN) { + list_add_tail(&src->lru, ret); if (put_new_page) put_new_page(&dst->page, private); @@ -1564,7 +1599,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, enum migrate_mode mode, int reason, struct list_head *ret_folios, struct migrate_pages_stats *stats) { - int retry = 1; + int retry; int large_retry = 1; int thp_retry = 1; int nr_failed = 0; @@ -1573,13 +1608,19 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, int pass = 0; bool is_large = false; bool is_thp = false; - struct folio *folio, *folio2, *dst = NULL; - int rc, nr_pages; + struct folio *folio, *folio2, *dst = NULL, *dst2; + int rc, rc_saved, nr_pages; LIST_HEAD(split_folios); + LIST_HEAD(unmap_folios); + LIST_HEAD(dst_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); bool no_split_folio_counting = false; + bool force_lock; -split_folio_migration: +retry: + rc_saved = 0; + force_lock = true; + retry = 1; for (pass = 0; pass < 10 && (retry || large_retry); pass++) { retry = 0; large_retry = 0; @@ -1599,16 +1640,15 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, cond_resched(); rc = migrate_folio_unmap(get_new_page, put_new_page, private, - folio, &dst, pass > 2, mode, - reason, ret_folios); - if (rc == MIGRATEPAGE_UNMAP) - rc = migrate_folio_move(put_new_page, private, - folio, dst, mode, - reason, ret_folios); + folio, &dst, pass > 2, force_lock, + mode, reason, ret_folios); /* * The rules are: * Success: folio will be freed + * Unmap: folio will be put on unmap_folios list, + * dst folio put on dst_folios list * -EAGAIN: stay on the from list + * -EDEADLOCK: stay on the from list * -ENOMEM: stay on the from list * -ENOSYS: stay on the from list * Other errno: put on ret_folios list @@ -1643,7 +1683,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, case -ENOMEM: /* * When memory is low, don't bother to try to migrate - * other folios, just exit. + * other folios, move unmapped folios, then exit. */ if (is_large) { nr_large_failed++; @@ -1682,7 +1722,14 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, /* nr_failed isn't updated for not used */ nr_large_failed += large_retry; stats->nr_thp_failed += thp_retry; - goto out; + rc_saved = rc; + if (list_empty(&unmap_folios)) + goto out; + else + goto move; + case -EDEADLOCK: + rc_saved = rc; + goto move; case -EAGAIN: if (is_large) { large_retry++; @@ -1696,6 +1743,15 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, stats->nr_succeeded += nr_pages; stats->nr_thp_succeeded += is_thp; break; + case MIGRATEPAGE_UNMAP: + /* + * We have locked some pages, don't force lock + * to avoid deadlock. + */ + force_lock = false; + list_move_tail(&folio->lru, &unmap_folios); + list_add_tail(&dst->lru, &dst_folios); + break; default: /* * Permanent failure (-EBUSY, etc.): @@ -1719,12 +1775,93 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, nr_large_failed += large_retry; stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; +move: + retry = 1; + for (pass = 0; pass < 10 && (retry || large_retry); pass++) { + retry = 0; + large_retry = 0; + thp_retry = 0; + nr_retry_pages = 0; + + dst = list_first_entry(&dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { + is_large = folio_test_large(folio); + is_thp = is_large && folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + + cond_resched(); + + rc = migrate_folio_move(put_new_page, private, + folio, dst, mode, + reason, ret_folios); + /* + * The rules are: + * Success: folio will be freed + * -EAGAIN: stay on the unmap_folios list + * Other errno: put on ret_folios list + */ + switch(rc) { + case -EAGAIN: + if (is_large) { + large_retry++; + thp_retry += is_thp; + } else if (!no_split_folio_counting) { + retry++; + } + nr_retry_pages += nr_pages; + break; + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + break; + default: + if (is_large) { + nr_large_failed++; + stats->nr_thp_failed += is_thp; + } else if (!no_split_folio_counting) { + nr_failed++; + } + + stats->nr_failed_pages += nr_pages; + break; + } + dst = dst2; + dst2 = list_next_entry(dst, lru); + } + } + nr_failed += retry; + nr_large_failed += large_retry; + stats->nr_thp_failed += thp_retry; + stats->nr_failed_pages += nr_retry_pages; + + if (rc_saved) + rc = rc_saved; + else + rc = nr_failed + nr_large_failed; +out: + /* Cleanup remaining folios */ + dst = list_first_entry(&dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { + int page_was_mapped = 0; + struct anon_vma *anon_vma = NULL; + + __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); + migrate_folio_undo_src(folio, page_was_mapped, anon_vma, + ret_folios); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, put_new_page, private); + dst = dst2; + dst2 = list_next_entry(dst, lru); + } + /* * Try to migrate split folios of fail-to-migrate large folios, no * nr_failed counting in this round, since all split folios of a * large folio is counted as 1 failure in the first round. */ - if (!list_empty(&split_folios)) { + if (rc >= 0 && !list_empty(&split_folios)) { /* * Move non-migrated folios (after 10 retries) to ret_folios * to avoid migrating them again. @@ -1732,12 +1869,16 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, list_splice_init(from, ret_folios); list_splice_init(&split_folios, from); no_split_folio_counting = true; - retry = 1; - goto split_folio_migration; + goto retry; } - rc = nr_failed + nr_large_failed; -out: + /* + * We have unlocked all locked pages, so we can force lock now, let's + * try again. + */ + if (rc == -EDEADLOCK) + goto retry; + return rc; } From patchwork Tue Dec 27 00:28:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13081976 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCCDBC4332F for ; Tue, 27 Dec 2022 00:29:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6895294000C; Mon, 26 Dec 2022 19:29:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 63900940007; Mon, 26 Dec 2022 19:29:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5014E94000C; Mon, 26 Dec 2022 19:29:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 414DC940007 for ; Mon, 26 Dec 2022 19:29:49 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1EB2A140653 for ; Tue, 27 Dec 2022 00:29:49 +0000 (UTC) X-FDA: 80286203298.09.9064536 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf19.hostedemail.com (Postfix) with ESMTP id 33C561A0007 for ; Tue, 27 Dec 2022 00:29:47 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=FuOQiOXk; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672100987; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eLdqYltBVLujlBa34RffV+DhEK2QGJf9sLFKaJk8IOI=; b=wA1Riul5JfUxKmJ9fvvfgb2faLRKOjIhlMrl6m1GNKy/gpHzR82TN6GEQDglMA27iU7CvD Vg6BWcbrtWHXU261G01sdMiOZgodojQejEOcS0LNy/77sfDG2gjrsGcEziRxaGHiDgV//G e2NU9CK61xp/mhGME21ZBy6aiwi0sxM= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=FuOQiOXk; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672100987; a=rsa-sha256; cv=none; b=vVtmmq8Cpc9EuD226lF+wKrdqaI/KHKNTC9vu7zhFTLEPsAmwWBDa/9/RP9wnC4kiRcii2 BL9CqktZBQpu//GBCBweaA6pFpTVVr2P8DUOaQMvrHNmWD2zEGBjC3CUM9o5IMY2+k3p4c 6pXp4Ar7/SyVdIq9tjR7qnIJz5QvH8U= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672100987; x=1703636987; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Vd3x7uf76BZx4AuXVaHyrVdNIQHfuHN+j/7ggUNEXew=; b=FuOQiOXknbxkHHl5wg2Cjic+Ln6l1+wZLZzV5nQshTYJesSKn1Azz0rt YeVyfRLh5dw0HMykHtMd9sYco8W7dXqNzFTONsjdyGchbNbtYw/H2nS1y RSg57LghW43pc/1PgSu7DF2OGbhMmtYrar+RVmMHS3g+TDo74yYBxEfwN a/KFnh+5SqYhqJa8sBB1nb5So4Gxt4bxlvQe6wjazHnoaQ9+DnQx0lavh JAF/t4GAFEsBQ8FTmAFQ7EfiEVeVTVeFdkTb/c/l5ANK00FVSfvCO5KbX cVxkyD3OnQSZzhiywtc9XOVVvDaBo1SdfA/lo6n6NYB3eSFUkwcuKewCe A==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="322597267" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="322597267" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:46 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="760172236" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="760172236" Received: from yyang3-mobl1.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.212.104]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:42 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH 6/8] migrate_pages: move migrate_folio_done() and migrate_folio_unmap() Date: Tue, 27 Dec 2022 08:28:57 +0800 Message-Id: <20221227002859.27740-7-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221227002859.27740-1-ying.huang@intel.com> References: <20221227002859.27740-1-ying.huang@intel.com> MIME-Version: 1.0 X-Stat-Signature: d9743wye95z9airo1wxj7fzd14geumw1 X-Rspam-User: X-Rspamd-Queue-Id: 33C561A0007 X-Rspamd-Server: rspam06 X-HE-Tag: 1672100987-363131 X-HE-Meta: U2FsdGVkX19IFLfzNhUQMNFQC0qoGaoa8BWWFxs2uCAXioE5iZ9g7FvZ8EcclBP+hL3cyQDdZck9j946rQQFVBwYiN1mb3jYupRsNTLhwf3KnSmBK5cfUu5X+s/E1WLs2ziS+wL9FgKDCjCc4FFrFslwCp2/0XAE4sVi+ys+f7LZbtcvfbR3NrIe0t0NcOP8DVWAxnqNuwDSnTnd0kkyWggW85MKqfJvRZY57o9dUw606ScZXkifQzGilW4afJ/1f6SqML+hwheOeFQTaJtYZs+tcMi0G8koaBqsitDCVOtLh/YRjC95FO6rzvX7OsSkoXp8A5eGNDJ50yu+GrsSMIRD7hEkMsTtMTEicVRmTUQkwh1qYAoBxG37CSzDZdX5p9LKZO56n7PDSdHgXpUpTmslVDZkXyRsUBIVr8M3QcVmioGwKAna+mrFEYcu6yBkHJaPPtJgvs9M4kD9uBj66w94hMO23Dw7zlzdWkGj5EfrqwIjefdQscoAucDEO/jaCg9oqh0hXo5sbh7WpGjx2PeCVABfVbgtd/ddTAAvtOVZ8fXs5xQG7xxrqla05PrIK4tTxXzWg5q/w0C+/hsDUdsSGclxoNrfM1hYej5qC9CBOmdKpWDa4utceGZukDl1MpXCJ17WlCIILNsT8rsm8trf1JDUmiIR7ckWHx4zvCRy5cJiH8IMQujmwR8ZMzqEchBd829q3noF54XKaubl4eQrl3AlzPx36Pco3qlg+sc9IMfXerb8Q7hF2m59VHHbedCvgDyPafeH+JCQKBSLtcEB/Ufxk3RI8fAjJ/AdtMtujwXYuN2qG0eQx8cSsE9R/hynD2nbLfI8Oj2TB4H561WC16tAYViGUaNmoC8u17z5GY8b8pXqLgxv9ISdwnR/7bsGYJTU3nsKr5FD1ase4gKAU1Vu45eUf5M+IVgj5KUu2i3mCKOHOYFBv0iafL8Pg+9ftcuRl58= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Just move the position of 2 functions. There's no any functionality change. This is to make it easier to review the next patch via putting code near its position in the next patch. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin --- mm/migrate.c | 136 +++++++++++++++++++++++++-------------------------- 1 file changed, 68 insertions(+), 68 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index dd68c3de3da8..70b987391296 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1051,6 +1051,23 @@ static void migrate_folio_undo_dst(struct folio *dst, folio_put(dst); } +static void migrate_folio_done(struct folio *src, + enum migrate_reason reason) +{ + /* + * Compaction can migrate also non-LRU pages which are + * not accounted to NR_ISOLATED_*. They can be recognized + * as __PageMovable + */ + if (likely(!__folio_test_movable(src))) + mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON + + folio_is_file_lru(src), -folio_nr_pages(src)); + + if (reason != MR_MEMORY_FAILURE) + /* We release the page in page_handle_poison. */ + folio_put(src); +} + static int __migrate_folio_unmap(struct folio *src, struct folio *dst, int force, bool force_lock, enum migrate_mode mode) { @@ -1186,6 +1203,57 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, return rc; } +/* Obtain the lock on page, remove all ptes. */ +static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, + unsigned long private, struct folio *src, + struct folio **dstp, int force, bool force_lock, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) +{ + struct folio *dst; + int rc = MIGRATEPAGE_UNMAP; + struct page *newpage = NULL; + + if (!thp_migration_supported() && folio_test_transhuge(src)) + return -ENOSYS; + + if (folio_ref_count(src) == 1) { + /* Folio was freed from under us. So we are done. */ + folio_clear_active(src); + folio_clear_unevictable(src); + /* free_pages_prepare() will clear PG_isolated. */ + list_del(&src->lru); + migrate_folio_done(src, reason); + return MIGRATEPAGE_SUCCESS; + } + + newpage = get_new_page(&src->page, private); + if (!newpage) + return -ENOMEM; + dst = page_folio(newpage); + *dstp = dst; + + dst->private = NULL; + rc = __migrate_folio_unmap(src, dst, force, force_lock, mode); + if (rc == MIGRATEPAGE_UNMAP) + return rc; + + /* + * A page that has not been migrated will have kept its + * references and be restored. + */ + /* restore the folio to right list. */ + if (rc != -EAGAIN && rc != -EDEADLOCK) + list_move_tail(&src->lru, ret); + + if (put_new_page) + put_new_page(&dst->page, private); + else + folio_put(dst); + + return rc; +} + static int __migrate_folio_move(struct folio *src, struct folio *dst, enum migrate_mode mode) { @@ -1239,74 +1307,6 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, return rc; } -static void migrate_folio_done(struct folio *src, - enum migrate_reason reason) -{ - /* - * Compaction can migrate also non-LRU pages which are - * not accounted to NR_ISOLATED_*. They can be recognized - * as __PageMovable - */ - if (likely(!__folio_test_movable(src))) - mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON + - folio_is_file_lru(src), -folio_nr_pages(src)); - - if (reason != MR_MEMORY_FAILURE) - /* We release the page in page_handle_poison. */ - folio_put(src); -} - -/* Obtain the lock on page, remove all ptes. */ -static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, - unsigned long private, struct folio *src, - struct folio **dstp, int force, bool force_lock, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) -{ - struct folio *dst; - int rc = MIGRATEPAGE_UNMAP; - struct page *newpage = NULL; - - if (!thp_migration_supported() && folio_test_transhuge(src)) - return -ENOSYS; - - if (folio_ref_count(src) == 1) { - /* Folio was freed from under us. So we are done. */ - folio_clear_active(src); - folio_clear_unevictable(src); - /* free_pages_prepare() will clear PG_isolated. */ - list_del(&src->lru); - migrate_folio_done(src, reason); - return MIGRATEPAGE_SUCCESS; - } - - newpage = get_new_page(&src->page, private); - if (!newpage) - return -ENOMEM; - dst = page_folio(newpage); - *dstp = dst; - - dst->private = NULL; - rc = __migrate_folio_unmap(src, dst, force, force_lock, mode); - if (rc == MIGRATEPAGE_UNMAP) - return rc; - - /* - * A page that has not been migrated will have kept its - * references and be restored. - */ - /* restore the folio to right list. */ - if (rc != -EAGAIN && rc != -EDEADLOCK) - list_move_tail(&src->lru, ret); - - if (put_new_page) - put_new_page(&dst->page, private); - else - folio_put(dst); - - return rc; -} - /* Migrate the folio to the newly allocated folio in dst. */ static int migrate_folio_move(free_page_t put_new_page, unsigned long private, struct folio *src, struct folio *dst, From patchwork Tue Dec 27 00:28:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13081977 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39EE5C4332F for ; Tue, 27 Dec 2022 00:29:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA40494000D; Mon, 26 Dec 2022 19:29:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C7B04940007; Mon, 26 Dec 2022 19:29:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B1B8094000D; Mon, 26 Dec 2022 19:29:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A096A940007 for ; Mon, 26 Dec 2022 19:29:53 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7B4721A0561 for ; Tue, 27 Dec 2022 00:29:53 +0000 (UTC) X-FDA: 80286203466.11.F7F64CB Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf19.hostedemail.com (Postfix) with ESMTP id 8B5621A000A for ; Tue, 27 Dec 2022 00:29:51 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=dM2AEDPX; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672100991; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=obHVx6mzXRlDp3zjrW+S8BR5YEyinWeO9SGIbjTLGXo=; b=zL0PMjIKwDrV11FeZpr6olRaPo2u7QkMwWkqlory1GhGEkG9Z3C3nsSFncf71ZhYygxBed Ypt7Z0XQ6pT90PA3O8XbRVd0lDlAIAsSsWJKsA/q9Zwv8B6mn4EXpD1Hhlvrq158sgGD9N Z8bC1WznkpHoMDWpWk4mTSFQK7laCbc= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=dM2AEDPX; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672100991; a=rsa-sha256; cv=none; b=Ell+FTTjl6ZrrDq6XHsakiXJ28YJFLSfYagmSF6enmGJcxPAJ2uuC4c8DEfIYbWZdEftWJ 2z4q5p0/oPOgGfQg5WngnYOgeOyhQDuTRbNCNZK/ixCu4QbfkKIGmJuwNIsWklVGnMDumd cQl7UjFi7767+mSDUw70apzpLGOqBNs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672100991; x=1703636991; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=pgO5oJrN4cdhAhS4PST0kGA04r/Qbf7dMr/JbfQPcJw=; b=dM2AEDPXUMzxcjEhG55nDXHmHfYNacHBdN9UpL+2PH37Ygac2gTs827M qAMjFVOoV6bTqAcuzUkpug/NVhOaSv9o4xoOEdF4tcSUk44o+eAqTdVAD 0oTtfnUphHsYVQM17Re2DB8dh3/+E9mopZlPCoWHKzb3fBjTdb+IoBH02 azsjoucRsf0IytGuPaWDMahWaafYrUHf1cH40yVuPpxYq9YdNzoq0RUxX A6ba/vzpLVf55HSkD3B7/l4XoNPlFURgIEMEAE93iM/iOmCC1IaTOJvGt 2VjABxV46D9HhiBfZWpTwXr3fUXX6aBM2Y7pAPsiVRqT6y/4cUPUJv8GD g==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="322597279" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="322597279" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:51 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="760172244" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="760172244" Received: from yyang3-mobl1.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.212.104]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:47 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH 7/8] migrate_pages: share more code between _unmap and _move Date: Tue, 27 Dec 2022 08:28:58 +0800 Message-Id: <20221227002859.27740-8-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221227002859.27740-1-ying.huang@intel.com> References: <20221227002859.27740-1-ying.huang@intel.com> MIME-Version: 1.0 X-Stat-Signature: qwhp31k5koq5yt3gazcx15z9zmgjdaki X-Rspam-User: X-Rspamd-Queue-Id: 8B5621A000A X-Rspamd-Server: rspam06 X-HE-Tag: 1672100991-427126 X-HE-Meta: U2FsdGVkX1/+5ny/R/FXTh1Jh9sNSVOlwgd+Y3psm2lLEMQ/McQJL6IoyOk0DR2QK2gJMWdXvMa7Tu4DITCMEMDA9JHPJ7h1+U+IZGF+YFjUltkoJHz9pYawxOTuwRt4GDe4LRFCTI+2vXPAFrWnaoBp8w0gqAgZmYdZ7VPNkfgeQHyB3rQTYqnCLjfdFQgudIJX3vVdH2dZ8tT6r65cRMtN/jH6zBnCl98p+UL0cCXqlXDuJ/oizXAozJu310HVSa3CT0M72N6xa1RciF1R/wbY761ZcQMUdFo2mspHworDny/3UH4X4BYkOVnLwd/jbpAwl6e7YK0gZ0zHYGNIfDcf7/yaJ5AvD3Ukd21/i6R4OXF2hUrtQiOU4rt7VpEwoaBJS/hRc8CGQdiycREcvHHZsXzJvzUjjJJVE/P5H2rtSi7/u1RwedZjpjGijhbnbc0HTFGQPQb9QRSkKBXkE0iFeZum2fy7FhamgeDB1oNcBTxhPqXI4OfOqnO18YTPxueFaiSHkBLgKqZzoyon2fG11smybsTay9NentjnjKLCwD7bazUw1BMsDizAii9T581WK3P7p77mSuP7UPz/5T3+ucbiHHnvvOD8CA/GCKcIgdh+MkoyL/HU/RxY3kDo9uCpbCUjg/vSRZRqDh2LlAiotrP0IfSF4S38YXeZV+VRFm5ssnCvVx9seuPe5o4/9HWwWjrazbSYc21y70BWQUsbK8OgF1DGmCs7DgrCTwkfNPnRW8m/3cDMtdOc+quA/iLVmj2xiz7EDVCY64go8izzmGN4AqtLXuMKzNEiLPqLogjDpApVsiPJ/6+lFRiUWDbOCNvwIlR9BYifDAgikRfVAPV0maFowczP6AA8o/D55Y8Ea6+FRW9bLTMZQt2SD7z2B+Hhwo1dA9MSCM8JwqYjy6IG6fAWye6ERMwsySys+AiI3yxrBxvmgzIWOTFNZuS2URpu9V0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a code cleanup patch to reduce the duplicated code between the _unmap and _move stages of migrate_pages(). No functionality change is expected. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin --- mm/migrate.c | 208 ++++++++++++++++++++------------------------------- 1 file changed, 82 insertions(+), 126 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 70b987391296..70a40b8fee1f 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1030,21 +1030,26 @@ static void __migrate_folio_extract(struct folio *dst, static void migrate_folio_undo_src(struct folio *src, int page_was_mapped, struct anon_vma *anon_vma, + bool locked, struct list_head *ret) { if (page_was_mapped) remove_migration_ptes(src, src, false); if (anon_vma) put_anon_vma(anon_vma); - folio_unlock(src); - list_move_tail(&src->lru, ret); + if (locked) + folio_unlock(src); + if (ret) + list_move_tail(&src->lru, ret); } static void migrate_folio_undo_dst(struct folio *dst, + bool locked, free_page_t put_new_page, unsigned long private) { - folio_unlock(dst); + if (locked) + folio_unlock(dst); if (put_new_page) put_new_page(&dst->page, private); else @@ -1068,14 +1073,44 @@ static void migrate_folio_done(struct folio *src, folio_put(src); } -static int __migrate_folio_unmap(struct folio *src, struct folio *dst, - int force, bool force_lock, enum migrate_mode mode) +/* Obtain the lock on page, remove all ptes. */ +static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, + unsigned long private, struct folio *src, + struct folio **dstp, int force, bool force_lock, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) { - int rc = -EAGAIN; + struct folio *dst; + int rc = MIGRATEPAGE_UNMAP; + struct page *newpage = NULL; int page_was_mapped = 0; struct anon_vma *anon_vma = NULL; bool is_lru = !__PageMovable(&src->page); + bool locked = false; + bool dst_locked = false; + + if (!thp_migration_supported() && folio_test_transhuge(src)) + return -ENOSYS; + + if (folio_ref_count(src) == 1) { + /* Folio was freed from under us. So we are done. */ + folio_clear_active(src); + folio_clear_unevictable(src); + /* free_pages_prepare() will clear PG_isolated. */ + list_del(&src->lru); + migrate_folio_done(src, reason); + return MIGRATEPAGE_SUCCESS; + } + + newpage = get_new_page(&src->page, private); + if (!newpage) + return -ENOMEM; + dst = page_folio(newpage); + *dstp = dst; + + dst->private = NULL; + rc = -EAGAIN; if (!folio_trylock(src)) { if (!force || mode == MIGRATE_ASYNC) goto out; @@ -1103,6 +1138,7 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, folio_lock(src); } + locked = true; if (folio_test_writeback(src)) { /* @@ -1117,10 +1153,10 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, break; default: rc = -EBUSY; - goto out_unlock; + goto out; } if (!force) - goto out_unlock; + goto out; folio_wait_writeback(src); } @@ -1150,7 +1186,8 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, * This is much like races on refcount of oldpage: just don't BUG(). */ if (unlikely(!folio_trylock(dst))) - goto out_unlock; + goto out; + dst_locked = true; if (unlikely(!is_lru)) { __migrate_folio_record(dst, page_was_mapped, anon_vma); @@ -1172,7 +1209,7 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, if (!src->mapping) { if (folio_test_private(src)) { try_to_free_buffers(src); - goto out_unlock_both; + goto out; } } else if (folio_mapped(src)) { /* Establish migration ptes */ @@ -1187,75 +1224,27 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, return MIGRATEPAGE_UNMAP; } - - if (page_was_mapped) - remove_migration_ptes(src, src, false); - -out_unlock_both: - folio_unlock(dst); -out_unlock: - /* Drop an anon_vma reference if we took one */ - if (anon_vma) - put_anon_vma(anon_vma); - folio_unlock(src); out: - - return rc; -} - -/* Obtain the lock on page, remove all ptes. */ -static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, - unsigned long private, struct folio *src, - struct folio **dstp, int force, bool force_lock, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) -{ - struct folio *dst; - int rc = MIGRATEPAGE_UNMAP; - struct page *newpage = NULL; - - if (!thp_migration_supported() && folio_test_transhuge(src)) - return -ENOSYS; - - if (folio_ref_count(src) == 1) { - /* Folio was freed from under us. So we are done. */ - folio_clear_active(src); - folio_clear_unevictable(src); - /* free_pages_prepare() will clear PG_isolated. */ - list_del(&src->lru); - migrate_folio_done(src, reason); - return MIGRATEPAGE_SUCCESS; - } - - newpage = get_new_page(&src->page, private); - if (!newpage) - return -ENOMEM; - dst = page_folio(newpage); - *dstp = dst; - - dst->private = NULL; - rc = __migrate_folio_unmap(src, dst, force, force_lock, mode); - if (rc == MIGRATEPAGE_UNMAP) - return rc; - /* * A page that has not been migrated will have kept its * references and be restored. */ /* restore the folio to right list. */ - if (rc != -EAGAIN && rc != -EDEADLOCK) - list_move_tail(&src->lru, ret); + if (rc == -EAGAIN || rc == -EDEADLOCK) + ret = NULL; - if (put_new_page) - put_new_page(&dst->page, private); - else - folio_put(dst); + migrate_folio_undo_src(src, page_was_mapped, anon_vma, locked, ret); + if (dst) + migrate_folio_undo_dst(dst, dst_locked, put_new_page, private); return rc; } -static int __migrate_folio_move(struct folio *src, struct folio *dst, - enum migrate_mode mode) +/* Migrate the folio to the newly allocated folio in dst. */ +static int migrate_folio_move(free_page_t put_new_page, unsigned long private, + struct folio *src, struct folio *dst, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) { int rc; int page_was_mapped = 0; @@ -1264,9 +1253,10 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); rc = move_to_new_folio(dst, src, mode); + if (rc) + goto out; - if (rc != -EAGAIN) - list_del(&dst->lru); + list_del(&dst->lru); /* * When successful, push dst to LRU immediately: so that if it * turns out to be an mlocked page, remove_migration_ptes() will @@ -1276,74 +1266,40 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, * unsuccessful, and other cases when a page has been temporarily * isolated from the unevictable LRU: but this case is the easiest. */ - if (rc == MIGRATEPAGE_SUCCESS) { - folio_add_lru(dst); - if (page_was_mapped) - lru_add_drain(); - } - - if (rc == -EAGAIN) { - __migrate_folio_record(dst, page_was_mapped, anon_vma); - return rc; - } - + folio_add_lru(dst); if (page_was_mapped) - remove_migration_ptes(src, - rc == MIGRATEPAGE_SUCCESS ? dst : src, false); + lru_add_drain(); + if (page_was_mapped) + remove_migration_ptes(src, dst, false); folio_unlock(dst); - /* Drop an anon_vma reference if we took one */ - if (anon_vma) - put_anon_vma(anon_vma); - folio_unlock(src); + set_page_owner_migrate_reason(&dst->page, reason); /* * If migration is successful, decrease refcount of dst, * which will not free the page because new page owner increased * refcounter. */ - if (rc == MIGRATEPAGE_SUCCESS) - folio_put(dst); - - return rc; -} - -/* Migrate the folio to the newly allocated folio in dst. */ -static int migrate_folio_move(free_page_t put_new_page, unsigned long private, - struct folio *src, struct folio *dst, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) -{ - int rc; - - rc = __migrate_folio_move(src, dst, mode); - if (rc == MIGRATEPAGE_SUCCESS) - set_page_owner_migrate_reason(&dst->page, reason); - - if (rc != -EAGAIN) { - /* - * A folio that has been migrated has all references - * removed and will be freed. A folio that has not been - * migrated will have kept its references and be restored. - */ - list_del(&src->lru); - } + folio_put(dst); /* - * If migration is successful, releases reference grabbed during - * isolation. Otherwise, restore the folio to right list unless - * we want to retry. + * A page that has been migrated has all references removed + * and will be freed. */ - if (rc == MIGRATEPAGE_SUCCESS) { - migrate_folio_done(src, reason); - } else if (rc != -EAGAIN) { - list_add_tail(&src->lru, ret); + list_del(&src->lru); + migrate_folio_undo_src(src, 0, anon_vma, true, NULL); + migrate_folio_done(src, reason); - if (put_new_page) - put_new_page(&dst->page, private); - else - folio_put(dst); + return rc; +out: + if (rc == -EAGAIN) { + __migrate_folio_record(dst, page_was_mapped, anon_vma); + return rc; } + migrate_folio_undo_src(src, page_was_mapped, anon_vma, true, ret); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, true, put_new_page, private); + return rc; } @@ -1849,9 +1805,9 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); migrate_folio_undo_src(folio, page_was_mapped, anon_vma, - ret_folios); + true, ret_folios); list_del(&dst->lru); - migrate_folio_undo_dst(dst, put_new_page, private); + migrate_folio_undo_dst(dst, true, put_new_page, private); dst = dst2; dst2 = list_next_entry(dst, lru); } From patchwork Tue Dec 27 00:28:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13081978 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B634C4332F for ; Tue, 27 Dec 2022 00:29:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D3F4194000E; Mon, 26 Dec 2022 19:29:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CF03D940007; Mon, 26 Dec 2022 19:29:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB7F194000E; Mon, 26 Dec 2022 19:29:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id ACD54940007 for ; Mon, 26 Dec 2022 19:29:57 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8E585808CD for ; Tue, 27 Dec 2022 00:29:57 +0000 (UTC) X-FDA: 80286203634.12.001F0DB Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf19.hostedemail.com (Postfix) with ESMTP id AC3BD1A0016 for ; Tue, 27 Dec 2022 00:29:55 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ih8KNluk; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672100996; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SmzrhIzjhoWM5s5PJ10T2PWwXsxkQGmQGJgyNR100N8=; b=VbVsspkkjJWVo5YJneUmai8J8UqVglnMvELcyKojweuCM9o9/GgSrT+l3JARKyLsUZ5RHi hBQKvCxw/hsKW7wvijjNjS/LtcsJ5G9w8pEtsLZ2McU0jg+5Bp0XNR6auhIZecRwgZkb0O VxmFcj4jBr4/9QD6k2l8SzOjaleNmwg= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ih8KNluk; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672100996; a=rsa-sha256; cv=none; b=xA2sn6BbxFtHN85VZNamqrFfJFeF3cFvj1rREWqK99Spuf9pBfL+CbPaxR3f2hMQNqzi4S Umgswbzg+TqGgmDE+974TjXgXCkK4cSOdKs8k9T+Jkpes3m3Vlh4fB910/wR2XP3B9ff5S 9Qzpn3ZqJA5Ya27zcWgoN1bcthdd2Q0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672100995; x=1703636995; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ba35wqiPwZ1LzOMWrGbZ6/UPey9BiC9eCP+Pk93x5S8=; b=ih8KNlukDsKHZMdppIzuaC83WyoKvunXiIbcFMcuTPnzgGnoJyzbbe7Y OD7+cQ44lb7/5di70irgVQ8cb1emHtV23AkhHVoU817Hfn5fVmRMSGaye p3va+83ltL1ol2yJjiaCKkbuzvq0RgOFrkFNb0p91swLvoKc+NDEXDPby W8kFmhpeE9WRWcblPbdOf/y8I11LYfHP0dYDRovSvhaBgpZhlzORQRnBV r//Uv/8YT43q4I4nC4Un+HE95T2BTmMo4CFze3J7ibQkGWOYnHIZ324d4 Uy/+JSJQhfGQuJDJUC8URAsnlqdnSDtq6TPq/sTnPaqhpX7ZjuRiWqcL+ Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="322597290" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="322597290" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:55 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="760172251" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="760172251" Received: from yyang3-mobl1.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.212.104]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:51 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH 8/8] migrate_pages: batch flushing TLB Date: Tue, 27 Dec 2022 08:28:59 +0800 Message-Id: <20221227002859.27740-9-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221227002859.27740-1-ying.huang@intel.com> References: <20221227002859.27740-1-ying.huang@intel.com> MIME-Version: 1.0 X-Stat-Signature: ypwnzjyi98oxfu41t6kw9nun4bbnnm1w X-Rspam-User: X-Rspamd-Queue-Id: AC3BD1A0016 X-Rspamd-Server: rspam06 X-HE-Tag: 1672100995-166062 X-HE-Meta: U2FsdGVkX19oL26DmTQyj7ANrxv8urDyduSgbbyUpjAaveZMcCo8qw0PECy4DtaF7Tjuxu33Abdn4ewbyPIgXBVtAHlGrKtx9oj8mCzriICRkHfukrIVv9HctY24mGHebbIdR99Bg9m4SExcjMm/fzrLPQpZCZrrDFGE0GH/ZTH7LL5tdx/p5ldjBiEpo0ODP9h+A8o1w31Ble2R24pd2C8USBySQndwO7CHRKTlULWl/HX57kJteEMjs5FF2wQq6xhFLjTsBQbevcwpRaBNDhmKopOSfXZaP4A7IT5wDi0Yg3k+v+Z545q0/htk32wOmSluY3bpVSKw3aA/BAFMzR3pVA4M7XR6g9jbRLggLFd2BDgAe/ScRN+EAy8X2STSiTD1b49WSl48Mcq40DiMzSZiqznG/3uaFhazSO5HvdxoSiAZd1cijt7w5phFVeICTLfM9WNSYgIsd4Hlvz99o057VAxTOeCU5v/JMYPSvfjMbo0HBJGDruEq3hsg8raiDxQJt1Rr0hXq2WSV7+BGDvh3xLFPBLE20SO1PSxUNYFXIYnwTleqs479Nj738RI5bB7nJzlXVsG/KcIwRhRc+pJMe4v/d/lr+ub9JOXY+z35kG8CqnhEtlMXnNTWlxXdQufjY1Iv68hYl8xI8P1Ec77GKQ0UQHdwrkN3HLoq+6Sk96lFkQiUzm6bTus/btJe4QKuKJHx0eQvy9C3Qe+1QUaKx/qKMnSMUU+/PBiI7CmhYDREIvBCC8uYLmnvIQKGtWOpmtLjzMTRJzx+fOhgi4OiKDl0O7Jv4Od/yLhAfE7dCzcaSuPovKQ+YOiqaOkmt1W+YU0YsfE/gUGtU2vxQpM3sbonUakkoQ1CFzAbqK1JyhOi7qxV7EyaiQaMgXOqiBDt78BIIjfIGaFf3D6swnZ/1tY+1XpwlSiMRvX995bKkw6nAig/Pq0NWaDKUAJNseVtoCqXMXU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The TLB flushing will cost quite some CPU cycles during the folio migration in some situations. For example, when migrate a folio of a process with multiple active threads that run on multiple CPUs. After batching the _unmap and _move in migrate_pages(), the TLB flushing can be batched easily with the existing TLB flush batching mechanism. This patch implements that. We use the following test case to test the patch. On a 2-socket Intel server, - Run pmbench memory accessing benchmark - Run `migratepages` to migrate pages of pmbench between node 0 and node 1 back and forth. With the patch, the TLB flushing IPI reduces 99.1% during the test and the number of pages migrated successfully per second increases 291.7%. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin --- mm/migrate.c | 4 +++- mm/rmap.c | 20 +++++++++++++++++--- 2 files changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 70a40b8fee1f..d7413164e748 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1215,7 +1215,7 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page /* Establish migration ptes */ VM_BUG_ON_FOLIO(folio_test_anon(src) && !folio_test_ksm(src) && !anon_vma, src); - try_to_migrate(src, 0); + try_to_migrate(src, TTU_BATCH_FLUSH); page_was_mapped = 1; } @@ -1732,6 +1732,8 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; move: + try_to_unmap_flush(); + retry = 1; for (pass = 0; pass < 10 && (retry || large_retry); pass++) { retry = 0; diff --git a/mm/rmap.c b/mm/rmap.c index b616870a09be..2e125f3e462e 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1976,7 +1976,21 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, } else { flush_cache_page(vma, address, pte_pfn(*pvmw.pte)); /* Nuke the page table entry. */ - pteval = ptep_clear_flush(vma, address, pvmw.pte); + if (should_defer_flush(mm, flags)) { + /* + * We clear the PTE but do not flush so potentially + * a remote CPU could still be writing to the folio. + * If the entry was previously clean then the + * architecture must guarantee that a clear->dirty + * transition on a cached TLB entry is written through + * and traps if the PTE is unmapped. + */ + pteval = ptep_get_and_clear(mm, address, pvmw.pte); + + set_tlb_ubc_flush_pending(mm, pte_dirty(pteval)); + } else { + pteval = ptep_clear_flush(vma, address, pvmw.pte); + } } /* Set the dirty flag on the folio now the pte is gone. */ @@ -2148,10 +2162,10 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags) /* * Migration always ignores mlock and only supports TTU_RMAP_LOCKED and - * TTU_SPLIT_HUGE_PMD and TTU_SYNC flags. + * TTU_SPLIT_HUGE_PMD, TTU_SYNC, and TTU_BATCH_FLUSH flags. */ if (WARN_ON_ONCE(flags & ~(TTU_RMAP_LOCKED | TTU_SPLIT_HUGE_PMD | - TTU_SYNC))) + TTU_SYNC | TTU_BATCH_FLUSH))) return; if (folio_is_zone_device(folio) &&