From patchwork Mon Nov 13 17:01:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13454191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59BE1C4167D for ; Mon, 13 Nov 2023 17:02:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DACA68D0031; Mon, 13 Nov 2023 12:02:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D5C718D002E; Mon, 13 Nov 2023 12:02:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BAE448D0031; Mon, 13 Nov 2023 12:02:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A79DF8D002E for ; Mon, 13 Nov 2023 12:02:32 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7502B1A0864 for ; Mon, 13 Nov 2023 17:02:32 +0000 (UTC) X-FDA: 81453549744.26.9832C68 Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) by imf02.hostedemail.com (Postfix) with ESMTP id 6D3AD80026 for ; Mon, 13 Nov 2023 17:02:18 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=sent.com header.s=fm3 header.b=7HB3Wyi+; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=TPd9+Ccc; spf=pass (imf02.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.26 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1699894939; a=rsa-sha256; cv=none; b=yMx8rDdRhnn/1LkStbeFwweRTfxrLcsifVTKpD6QrVXj9Yfo5/kxIOSRK/HoevxLnsNW6x OajVKXfTsDXUZ8uuPAE/QwStEJqN5t1hv5djv9roOls/wHnsaeBFI52yfJsrhcIHK1u3kZ fzqGypuAkMf2Vdt4+fRd5RA8U3IIsro= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=sent.com header.s=fm3 header.b=7HB3Wyi+; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=TPd9+Ccc; spf=pass (imf02.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.26 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1699894939; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=urfl5JZUWyqykmUphyjWYkdFE+bafcBTkV8p8IuqwbA=; b=d02YxUS/XabDB8jSCBG/PhvKKnM80PrTU9GpfyAinCIbI63WGcvjKwFLmuw7HEwAWhH88f XElDyLjlYUCJWN3+7K2+Qvt87hjT9C7o4bKYHPBpC252oceDtmwWNqA4uBVaYs1CX2RmLs vNJN7gszobbrfeTSRk8BzvicvrKgtkQ= Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 010A05C01B7; Mon, 13 Nov 2023 12:02:18 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Mon, 13 Nov 2023 12:02:18 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:sender:subject:subject:to:to; s= fm3; t=1699894937; x=1699981337; bh=urfl5JZUWyqykmUphyjWYkdFE+ba fcBTkV8p8IuqwbA=; b=7HB3Wyi+1gRLdhpQriK/kyewm4aYNjtRZ2nu3IH7+wuu 2vsuB2HaPHYBy0ZZNHffjdKm+Sly7mnILjLxEcbrEPUIyF+rOoVwGNzTna0apkfD +EW9ORilVpQXmX2xDBPplisCDSM6hxnR/ESebHQJ6wGq7s6Wfa16M452P3I/ZqxV 44GnQKHspiQ8jZvk0d9h1oQ26cWjEqJ05V+obJDPgAoah0aDRtom2EVauTJ3i+HK 4JGcZ2r5vJF5Sjh/u2MK74H/Oun9AVv642CvBvYM0DwxwydVFu79uudd6FMdYddk dssWmmhtQpq619UGwR9+6re/dTW1KB+yuYUH6GFVMQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1699894937; x=1699981337; bh=urfl5JZUWyqykmUphyjWYkdFE+ba fcBTkV8p8IuqwbA=; b=TPd9+CccOdaGln4m8Tv8Duuxgf3e4Y8CQUAu137jPias LNS4P77nwQiCkh+2T11QwQgTHwXDMnrq1QN5d4NfoFs5pfxymEeMCH8LoxstLNNI mQF0karJJx7sg01u/LW5NDMXRTFVlSLOSMY7Oi7lrQWPOkSK3h5Duf34/2SI67TG W50qOEvqFUkbhtEHm+DxZJj/xueyu647kqMFBk0eqyYuq81C9B0skpBnBf0+5HIv dnk7GRsJ9KdEsJTihPFSZ3s8sd6ixsQOzQUVrsLJ6zWqPlnzz2EgvdrtMfSeKRH/ QpipWEiZpRAoigZs0yBHRGyaNzrsvlUA3k7VQQ5G5g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrudeftddgleefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhhrgggtgfesthekredtredtjeenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepje ekteekffelleekudfftdefvddtjeejuedtuedtteegjefgvedtfedujeekieevnecuvehl uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 13 Nov 2023 12:02:15 -0500 (EST) From: Zi Yan To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Zi Yan , "Huang, Ying" , Ryan Roberts , Andrew Morton , "Matthew Wilcox (Oracle)" , David Hildenbrand , "Yin, Fengwei" , Yu Zhao , Vlastimil Babka , "Kirill A . Shutemov" , Johannes Weiner , Baolin Wang , Kemeng Shi , Mel Gorman , Rohan Puri , Mcgrof Chamberlain , Adam Manzanares , "Vishal Moola (Oracle)" Subject: [PATCH v1 1/4] mm/compaction: enable compacting >0 order folios. Date: Mon, 13 Nov 2023 12:01:54 -0500 Message-ID: <20231113170157.280181-2-zi.yan@sent.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231113170157.280181-1-zi.yan@sent.com> References: <20231113170157.280181-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 6D3AD80026 X-Stat-Signature: pc36hpc6sgaouefof4j34m7kqr8rfwz9 X-Rspam-User: X-HE-Tag: 1699894938-3143 X-HE-Meta: U2FsdGVkX1/mJuoHePdn7EmVAIyzCIm2U7RDB+hk9iTmxJxcFlEpQ5AjdGGZWGoAZRzOH0Qw5ceMFh8U3o5mLACjAS+TiUfLSKH6hOQ00zM+83MG6VrdfqdmYVXJaj15oL16VIReqqfjdLG9kwPL7VLonolGhKH+NnkJBMgyYviNt1gxs6dU8CWsbuYYebr5PFPg5UuRaJHpmmya+vWYDZSQQJ1rnNorxMNlrbJHM8bPKnbHSSVmVfYfFjkvy7B+9dkz35+SZujVD8RZvcmAto06GzeKXCFi8HzBknTN5g0cDiO5qwtejpo+BJ6qcz3M7bQScAG7BBftwoYdaedNX4bZiRyItahgqChsuil9gztn84vx5XPMeXkGIC0m5J1bpYUPsawgVqb1AqhFOzuyqSZ9LtFeYC/Sgi8GLgrPoZxkPk3CDxVZpPqDTDw5XsdpOf9gbgrRWtKvePhd7tGUo5qnTO4pQezYcfVSEINxZjiv1KHV9C+fEHhTd2RmBn1dNR19oup2C5aWuBIfvLo2LQ3vSOE4J7A9E2GmSGZWy5JIcca4Dk+dW/4lNhReCzIVSbPHo9SMq77FbnoGHh0cfv1WUrzIF/aN4zED5LUHPpLYd21ONTY/nLLFu4yOkvs9yxm5ZIMM7gVm9Ylk3nTFQ7rKtxOyGlUVK5lZjOViLi74f0C+vkwBcU9bhOIhPmdvGGrNKSir9VfnSjJSzm2kAulWDW8pYilVovL9yewdG+YoeyDgsefYlX6QmJ98S9uidr9CuiAcL0NyApoigmq7oJcFbo4MX1MSVABxADn/vPe4XKbH4TkV2kGrGjWX1HhqbHf5G4dbxgKOLOC4hlk8DgpW3BqFzqUcKbeKcC6RiKddb7NMUENe1Krh3idrk33W+BTuwZbYqQO3P7SD2PEo3y3SpbwxSKBnshoFrgmrrEKmUooWd3UbWbbOUWdkyfO2WMI4qXYRfHzJq16Z2dF kY+dFLTe BJDVzjK0wT9TZvXtB1fVfBLd8t46KctmcSA7D+9Lncn2FYbpAHW8THtny3V6KOypY54rPHvVpirc6oqplurPdreqLM/Fj6f9TQsufBSr8qJmU0j4VQht67QS6YvN4WKuHq5Lapxc+44DtvVkQycZLZk5irvBfrKZVSnk3Gmw89/qS3RcW+YuJFstEsqqMrEPv3YbR8QM3XOod8mVVn0qT7cO2lXxX2Q8DvCafLPxqBf/Or1qsbRKEGSoeCtMS9rzeY07VRymXPDVpGLyNSsbyX7L5znOtMz/DqixCOKW2OghhAiOhYKTTtq+L7on9XEF1svqio+QddlbkF5hfJsigktindUyPjQnO/YY2NSd9CeXuqWx3W5n+ir5VheveViUugrdr7gTiS/2FY+NeuPR1oPvYwuLJKXFb6EBg4GNT/kSDfRDPKxfE9AZGDVWrNUWzO1YgFDxni3uwdSTQOHaFVaDORoaE5TBF0BgNxcbJcij6fQUGR4lfgMskU0H5j9538Gc4WPW4kTe6vKllZX+3nfq16qpJL9xLoVfLBU1JTGUJ818= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zi Yan migrate_pages() supports >0 order folio migration and during compaction, even if compaction_alloc() cannot provide >0 order free pages, migrate_pages() can split the source page and try to migrate the base pages from the split. It can be a baseline and start point for adding support for compacting >0 order folios. Suggested-by: Huang Ying Signed-off-by: Zi Yan --- mm/compaction.c | 57 ++++++++++++++++++++++++++++++++++++------------- 1 file changed, 42 insertions(+), 15 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 01ba298739dd..5217dd35b493 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -816,6 +816,21 @@ static bool too_many_isolated(struct compact_control *cc) return too_many; } +/* + * 1. if the page order is larger than or equal to target_order (i.e., + * cc->order and when it is not -1 for global compaction), skip it since + * target_order already indicates no free page with larger than target_order + * exists and later migrating it will most likely fail; + * + * 2. compacting > pageblock_order pages does not improve memory fragmentation, + * skip them; + */ +static bool skip_isolation_on_order(int order, int target_order) +{ + return (target_order != -1 && order >= target_order) || + order >= pageblock_order; +} + /** * isolate_migratepages_block() - isolate all migrate-able pages within * a single pageblock @@ -1009,7 +1024,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, /* * Regardless of being on LRU, compound pages such as THP and * hugetlbfs are not to be compacted unless we are attempting - * an allocation much larger than the huge page size (eg CMA). + * an allocation larger than the compound page size. * We can potentially save a lot of iterations if we skip them * at once. The check is racy, but we can consider only valid * values and the only danger is skipping too much. @@ -1017,11 +1032,18 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if (PageCompound(page) && !cc->alloc_contig) { const unsigned int order = compound_order(page); - if (likely(order <= MAX_ORDER)) { - low_pfn += (1UL << order) - 1; - nr_scanned += (1UL << order) - 1; + /* + * Skip based on page order and compaction target order + * and skip hugetlbfs pages. + */ + if (skip_isolation_on_order(order, cc->order) || + PageHuge(page)) { + if (order <= MAX_ORDER) { + low_pfn += (1UL << order) - 1; + nr_scanned += (1UL << order) - 1; + } + goto isolate_fail; } - goto isolate_fail; } /* @@ -1144,17 +1166,18 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, goto isolate_abort; } } + } - /* - * folio become large since the non-locked check, - * and it's on LRU. - */ - if (unlikely(folio_test_large(folio) && !cc->alloc_contig)) { - low_pfn += folio_nr_pages(folio) - 1; - nr_scanned += folio_nr_pages(folio) - 1; - folio_set_lru(folio); - goto isolate_fail_put; - } + /* + * Check LRU folio order under the lock + */ + if (unlikely(skip_isolation_on_order(folio_order(folio), + cc->order) && + !cc->alloc_contig)) { + low_pfn += folio_nr_pages(folio) - 1; + nr_scanned += folio_nr_pages(folio) - 1; + folio_set_lru(folio); + goto isolate_fail_put; } /* The folio is taken off the LRU */ @@ -1764,6 +1787,10 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data) struct compact_control *cc = (struct compact_control *)data; struct folio *dst; + /* this makes migrate_pages() split the source page and retry */ + if (folio_order(src) > 0) + return NULL; + if (list_empty(&cc->freepages)) { isolate_freepages(cc); From patchwork Mon Nov 13 17:01:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13454192 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A34A8C4332F for ; Mon, 13 Nov 2023 17:02:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 87D5A8D0032; Mon, 13 Nov 2023 12:02:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7B6FD8D002E; Mon, 13 Nov 2023 12:02:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 608E58D0032; Mon, 13 Nov 2023 12:02:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 4B7FD8D002E for ; Mon, 13 Nov 2023 12:02:33 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C08F31A0849 for ; Mon, 13 Nov 2023 17:02:30 +0000 (UTC) X-FDA: 81453549660.01.991571A Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) by imf08.hostedemail.com (Postfix) with ESMTP id 617D016005E for ; Mon, 13 Nov 2023 17:02:21 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=sent.com header.s=fm3 header.b=6vZ8QSc5; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=c4J28ZKi; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf08.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.26 as permitted sender) smtp.mailfrom=zi.yan@sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1699894941; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yXmXVOGVZ9680hBEB/GXlWgTDZKycYZ6tcV4N4qWe34=; b=d4Z0ak/7LZZuD3MOhheq0+iV9oDZzfcPGD10oroateIP/q9bFxUo+0YSFFQQ8koO24VZE3 KK4JSfiKYirmspSS4c4xStZ5VhakgOuTLlMC9aMZcTu5kwQH3+DksXM+OwU2Ln7K+DR4Ys xLfr6FdzSPi6QsOHwH1AmDE1rS2I2+8= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=sent.com header.s=fm3 header.b=6vZ8QSc5; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=c4J28ZKi; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf08.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.26 as permitted sender) smtp.mailfrom=zi.yan@sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1699894941; a=rsa-sha256; cv=none; b=AKM+ufLzcTRxn4Mrojgari0ayw6q9WqetBboIa2GkDtRrnapKctRKNTMY5fZoKpDY9Q0Mu 4O/eq19giQjgevypf17YLDiX9Vz2vYDBs7wXbFlFlPLc6xFuFIq98c+R4Ew0wsPX8cTY6R nZHhnlH7wum33gvaclvloUpl6BpVDC0= Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id AF85C5C01CA; Mon, 13 Nov 2023 12:02:20 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 13 Nov 2023 12:02:20 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:sender:subject:subject:to:to; s= fm3; t=1699894940; x=1699981340; bh=yXmXVOGVZ9680hBEB/GXlWgTDZKy cYZ6tcV4N4qWe34=; b=6vZ8QSc5QsL+6Ml1Z9LRB0cZF70TtpXwf0wyPq0Vvt36 vEKafKHGGxfNnO5VceD8pNj/FAwzvZZqm6QmaRS6E+VjbCMzODG7DZyvEj+746fp v7mbvCfnX73MXHwG/gfU1uJ4c4jR7eFswj9UuxhdTdV657W5gaOWSni6KXMmJXR0 uWklL13+NBaLhfKZnXXS3d+m4IYS5Se2IbX/2IfxlA1ORTA1/3Pxo8HoqJChEQvh ihYpzv0BWbk1SUTCfz40WtaNteTyaFLRHygdQAMxe5pXoDkV5F9h38jmpE4wZn7c Ws0N2doWqP8MoV3+9nau3c+u/msF8ghl7gUShR4Piw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1699894940; x=1699981340; bh=yXmXVOGVZ9680hBEB/GXlWgTDZKy cYZ6tcV4N4qWe34=; b=c4J28ZKitV8uAiFK6mlaD3Ie8/ukVuhMfqrVrig4P1nV +2b06RMcbpVGKQO+RIwvJZhc3QlcnlussY5xmK4gzfbns64/sOnEYm/EK0oWQw+u z+YuWFLqs+L+WXoPqmmTWK5FJxjd2WWaFrHNibVK+ow/TSrb6zhCazOyvH+WtRQ0 bdHY3m/1KteYv5CXZoDuyFBpnwJks05HvG/mosADk5lgh3ITxWRW8HmlYlPacoog HpLiI5rCRSkeqQm9VyIS753xeyNtzee3qwEY9GoSJFcXJ5AhJBfYHwnCs2e8Nymf jsm575obZbfzng0EFN/aH2WMWGD/g7uQPAHaygMGxA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrudeftddgleefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhhrgggtgfesthekredtredtjeenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepje ekteekffelleekudfftdefvddtjeejuedtuedtteegjefgvedtfedujeekieevnecuvehl uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 13 Nov 2023 12:02:18 -0500 (EST) From: Zi Yan To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Zi Yan , "Huang, Ying" , Ryan Roberts , Andrew Morton , "Matthew Wilcox (Oracle)" , David Hildenbrand , "Yin, Fengwei" , Yu Zhao , Vlastimil Babka , "Kirill A . Shutemov" , Johannes Weiner , Baolin Wang , Kemeng Shi , Mel Gorman , Rohan Puri , Mcgrof Chamberlain , Adam Manzanares , "Vishal Moola (Oracle)" Subject: [PATCH v1 2/4] mm/compaction: add support for >0 order folio memory compaction. Date: Mon, 13 Nov 2023 12:01:55 -0500 Message-ID: <20231113170157.280181-3-zi.yan@sent.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231113170157.280181-1-zi.yan@sent.com> References: <20231113170157.280181-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 617D016005E X-Stat-Signature: oitraguc9j7hhnum5z1x3ptphk3waj6q X-HE-Tag: 1699894941-966512 X-HE-Meta: U2FsdGVkX1+5n768PDT2K0sAKps/dp42z1cuubu+xw4knfz5T0Y4VPWLDkTJZDjcJQ101diF0gTaThA8NsgiS0gK5OCRB+Mk3kqzkaYshThNDoXxtmYwuVEfTxKhAlsmrLLo46FCyLeHEKwhVJJk2izGLXVc8vHUDs+AMmvijTsL0orXu1LSDDuJ8NtxN2YlCgbsIgS2jpqEoX4IPnRpQ9RIhE7MAAaCB+rjHOikOdpT3VqqGyxbpv6Y/ckBZNr3vC2/JbtH3sFhIJCmtym0n19h/qleiTStRJWnvQqnKcnHzsGhRbHZt7jeLMuEgsXYmK3y2nLfH1PzRlKu9jhZpmZMBtKjOOLDM0Qh20IkxKJsEENvxYWgTsSi90TqZdwD0c8P8Tsu7hHTeyf6OW8xkpTSzoRsiClYxC4dmkqdVgE4mZp6YrcShLhxtNgEC0eC40JqmaQHI5CTFUpWqkITeJMAcMnxzeih974KUhbpdHvzXQ95R7givXEBujeoyfmaJV8xZchfqragfO/1E/oc6IZOJJWNJxI1dlff4wBKU7x6ZKgGOhx/suR0fD2XBAH3dggtZqSfw0EfG7Vz8Ytq34PE43zVETuIA622MBJZgJSjuvC+OJDnUzwqFqo1wQpCV/9BkpXnnSMjAJrjMowSEvYwFmCjHSszQtwxB+uA982u9/K1+SI0DMldSuTK/SbPeNdt6hZ4XgynbuM+Uy0vceOT126lxcnUoSMa5nejA6tcL6Z0EB05t2SSQCd2pGaDYmxMBpZWDg+hQDirU7rCGuGCldbatEDA2Kn6lRIUXleZ7uwIgTV0r2nyx1ZCuFNPdUuM9eSMouN66rp30EHBh4W0EKV0XKYtXMLg6olCqIZW5VVbV1iJo9Jf6hlwWsK0ty0ka8UJXmsdTs2KtbvXEgOUj+VKvNoLcYH1Qg5xK1asvDRxIMH6RSQn0XV5L4sOwLDl+6yUkWgSwnuthr9 U2vyLZtD Y+yILvF2pMpQcvs6T6x2MuJt2CeyzKNTfAvZ4sIGwhffSxZhqBLcnC3VA4z+EaLxD5p3CrfrJwoWE15O/KsgMd7HO+UkswWcZtTeFSxpohNxFw1fT+WGVjTCLuauUOEyfIndYiQGDtB3MR4yLhJ8Fhfy16ZufyDlY9Y9FsOIoGQ+8sEJwnufsu0dCmXTFk2+s8zQgoBKZO5cqIU65BRELcTcOaTi9k1iSTICHHNRH6Kp+V6247cbR74M7MeykW4weD2bKsluemNLXFnc94CB/IhAHFfRPdPGlczjWOlWDpsQpJxuBhqD66mpcpLJJTVSWTGjDFYvh5BU1avFxdIc8xXxXZdqdm7/ggrZ1IXhqRWdlDKSUEjpD+hNnCxbYSQX+G4HOjZOVb7Rpvnbt8MH4nuJ4LCkwGldw9MWijAxtq46jGdv4iRo+UGEJIiOmW3AmwlR8RhHJVoKPaG41BDRa9U9nCb0aEKMKCHF1+48Bse9uTGME1gTJtZDwYw6rqgc4H74rT8qdRTEbf20= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zi Yan Before, memory compaction only migrates order-0 folios and skips >0 order folios. This commit adds support for >0 order folio compaction by keeping isolated free pages at their original size without splitting them into order-0 pages and using them directly during migration process. What is different from the prior implementation: 1. All isolated free pages are kept in a MAX_ORDER+1 array of page lists, where each page list stores free pages in the same order. 2. All free pages are not post_alloc_hook() processed nor buddy pages, although their orders are stored in first page's private like buddy pages. 3. During migration, in new page allocation time (i.e., in compaction_alloc()), free pages are then processed by post_alloc_hook(). When migration fails and a new page is returned (i.e., in compaction_free()), free pages are restored by reversing the post_alloc_hook() operations. Step 3 is done for a latter optimization that splitting and/or merging free pages during compaction becomes easier. Signed-off-by: Zi Yan --- mm/compaction.c | 160 ++++++++++++++++++++++++++++++------------------ mm/internal.h | 7 ++- 2 files changed, 108 insertions(+), 59 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 5217dd35b493..ec6b5cc7e907 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -66,45 +66,64 @@ static inline void count_compact_events(enum vm_event_item item, long delta) #define COMPACTION_HPAGE_ORDER (PMD_SHIFT - PAGE_SHIFT) #endif -static unsigned long release_freepages(struct list_head *freelist) +static void init_page_list(struct page_list *p) { - struct page *page, *next; - unsigned long high_pfn = 0; - - list_for_each_entry_safe(page, next, freelist, lru) { - unsigned long pfn = page_to_pfn(page); - list_del(&page->lru); - __free_page(page); - if (pfn > high_pfn) - high_pfn = pfn; - } - - return high_pfn; + INIT_LIST_HEAD(&p->pages); + p->nr_pages = 0; } -static void split_map_pages(struct list_head *list) +static void split_map_pages(struct page_list *freepages) { unsigned int i, order, nr_pages; struct page *page, *next; LIST_HEAD(tmp_list); - list_for_each_entry_safe(page, next, list, lru) { - list_del(&page->lru); + for (order = 0; order <= MAX_ORDER; order++) { + freepages[order].nr_pages = 0; + + list_for_each_entry_safe(page, next, &freepages[order].pages, lru) { + list_del(&page->lru); - order = page_private(page); - nr_pages = 1 << order; + nr_pages = 1 << order; - post_alloc_hook(page, order, __GFP_MOVABLE); - if (order) - split_page(page, order); + post_alloc_hook(page, order, __GFP_MOVABLE); + if (order) + split_page(page, order); - for (i = 0; i < nr_pages; i++) { - list_add(&page->lru, &tmp_list); - page++; + for (i = 0; i < nr_pages; i++) { + list_add(&page->lru, &tmp_list); + page++; + } + freepages[order].nr_pages += nr_pages; } + list_splice(&tmp_list, &freepages[order].pages); } +} - list_splice(&tmp_list, list); +static unsigned long release_free_list(struct page_list *freepages) +{ + int order; + unsigned long high_pfn = 0; + + for (order = 0; order <= MAX_ORDER; order++) { + struct page *page, *next; + + list_for_each_entry_safe(page, next, &freepages[order].pages, lru) { + unsigned long pfn = page_to_pfn(page); + + list_del(&page->lru); + /* + * Convert free pages into post allocation pages, so + * that we can free them via __free_page. + */ + post_alloc_hook(page, order, __GFP_MOVABLE); + __free_pages(page, order); + if (pfn > high_pfn) + high_pfn = pfn; + } + freepages[order].nr_pages = 0; + } + return high_pfn; } #ifdef CONFIG_COMPACTION @@ -583,7 +602,7 @@ static bool compact_unlock_should_abort(spinlock_t *lock, static unsigned long isolate_freepages_block(struct compact_control *cc, unsigned long *start_pfn, unsigned long end_pfn, - struct list_head *freelist, + struct page_list *freelist, unsigned int stride, bool strict) { @@ -657,7 +676,8 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, nr_scanned += isolated - 1; total_isolated += isolated; cc->nr_freepages += isolated; - list_add_tail(&page->lru, freelist); + list_add_tail(&page->lru, &freelist[order].pages); + freelist[order].nr_pages++; if (!strict && cc->nr_migratepages <= cc->nr_freepages) { blockpfn += isolated; @@ -722,7 +742,11 @@ isolate_freepages_range(struct compact_control *cc, unsigned long start_pfn, unsigned long end_pfn) { unsigned long isolated, pfn, block_start_pfn, block_end_pfn; - LIST_HEAD(freelist); + int order; + struct page_list tmp_freepages[MAX_ORDER + 1]; + + for (order = 0; order <= MAX_ORDER; order++) + init_page_list(&tmp_freepages[order]); pfn = start_pfn; block_start_pfn = pageblock_start_pfn(pfn); @@ -753,7 +777,7 @@ isolate_freepages_range(struct compact_control *cc, break; isolated = isolate_freepages_block(cc, &isolate_start_pfn, - block_end_pfn, &freelist, 0, true); + block_end_pfn, tmp_freepages, 0, true); /* * In strict mode, isolate_freepages_block() returns 0 if @@ -770,15 +794,15 @@ isolate_freepages_range(struct compact_control *cc, */ } - /* __isolate_free_page() does not map the pages */ - split_map_pages(&freelist); - if (pfn < end_pfn) { /* Loop terminated early, cleanup. */ - release_freepages(&freelist); + release_free_list(tmp_freepages); return 0; } + /* __isolate_free_page() does not map the pages */ + split_map_pages(tmp_freepages); + /* We don't use freelists for anything. */ return pfn; } @@ -1462,7 +1486,7 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn) if (!page) return; - isolate_freepages_block(cc, &start_pfn, end_pfn, &cc->freepages, 1, false); + isolate_freepages_block(cc, &start_pfn, end_pfn, cc->freepages, 1, false); /* Skip this pageblock in the future as it's full or nearly full */ if (start_pfn == end_pfn && !cc->no_set_skip_hint) @@ -1591,7 +1615,7 @@ static void fast_isolate_freepages(struct compact_control *cc) nr_scanned += nr_isolated - 1; total_isolated += nr_isolated; cc->nr_freepages += nr_isolated; - list_add_tail(&page->lru, &cc->freepages); + list_add_tail(&page->lru, &cc->freepages[order].pages); count_compact_events(COMPACTISOLATED, nr_isolated); } else { /* If isolation fails, abort the search */ @@ -1665,13 +1689,12 @@ static void isolate_freepages(struct compact_control *cc) unsigned long isolate_start_pfn; /* exact pfn we start at */ unsigned long block_end_pfn; /* end of current pageblock */ unsigned long low_pfn; /* lowest pfn scanner is able to scan */ - struct list_head *freelist = &cc->freepages; unsigned int stride; /* Try a small search of the free lists for a candidate */ fast_isolate_freepages(cc); if (cc->nr_freepages) - goto splitmap; + return; /* * Initialise the free scanner. The starting point is where we last @@ -1731,7 +1754,7 @@ static void isolate_freepages(struct compact_control *cc) /* Found a block suitable for isolating free pages from. */ nr_isolated = isolate_freepages_block(cc, &isolate_start_pfn, - block_end_pfn, freelist, stride, false); + block_end_pfn, cc->freepages, stride, false); /* Update the skip hint if the full pageblock was scanned */ if (isolate_start_pfn == block_end_pfn) @@ -1772,10 +1795,6 @@ static void isolate_freepages(struct compact_control *cc) * and the loop terminated due to isolate_start_pfn < low_pfn */ cc->free_pfn = isolate_start_pfn; - -splitmap: - /* __isolate_free_page() does not map the pages */ - split_map_pages(freelist); } /* @@ -1786,23 +1805,22 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data) { struct compact_control *cc = (struct compact_control *)data; struct folio *dst; + int order = folio_order(src); - /* this makes migrate_pages() split the source page and retry */ - if (folio_order(src) > 0) - return NULL; - - if (list_empty(&cc->freepages)) { + if (!cc->freepages[order].nr_pages) { isolate_freepages(cc); - - if (list_empty(&cc->freepages)) + if (!cc->freepages[order].nr_pages) return NULL; } - dst = list_entry(cc->freepages.next, struct folio, lru); + dst = list_first_entry(&cc->freepages[order].pages, struct folio, lru); + cc->freepages[order].nr_pages--; list_del(&dst->lru); - cc->nr_freepages--; - - return dst; + post_alloc_hook(&dst->page, order, __GFP_MOVABLE); + if (order) + prep_compound_page(&dst->page, order); + cc->nr_freepages -= 1 << order; + return page_rmappable_folio(&dst->page); } /* @@ -1813,9 +1831,34 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data) static void compaction_free(struct folio *dst, unsigned long data) { struct compact_control *cc = (struct compact_control *)data; + int order = folio_order(dst); + struct page *page = &dst->page; + + if (order) { + int i; - list_add(&dst->lru, &cc->freepages); - cc->nr_freepages++; + page[1].flags &= ~PAGE_FLAGS_SECOND; + for (i = 1; i < (1 << order); i++) { + page[i].mapping = NULL; + clear_compound_head(&page[i]); + page[i].flags &= ~PAGE_FLAGS_CHECK_AT_PREP; + } + + } + /* revert post_alloc_hook() operations */ + page->mapping = NULL; + page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; + set_page_count(page, 0); + page_mapcount_reset(page); + reset_page_owner(page, order); + page_table_check_free(page, order); + arch_free_page(page, order); + set_page_private(page, order); + INIT_LIST_HEAD(&dst->lru); + + list_add(&dst->lru, &cc->freepages[order].pages); + cc->freepages[order].nr_pages++; + cc->nr_freepages += 1 << order; } /* possible outcome of isolate_migratepages */ @@ -2439,6 +2482,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) const bool sync = cc->mode != MIGRATE_ASYNC; bool update_cached; unsigned int nr_succeeded = 0; + int order; /* * These counters track activities during zone compaction. Initialize @@ -2448,7 +2492,8 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) cc->total_free_scanned = 0; cc->nr_migratepages = 0; cc->nr_freepages = 0; - INIT_LIST_HEAD(&cc->freepages); + for (order = 0; order <= MAX_ORDER; order++) + init_page_list(&cc->freepages[order]); INIT_LIST_HEAD(&cc->migratepages); cc->migratetype = gfp_migratetype(cc->gfp_mask); @@ -2634,7 +2679,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) * so we don't leave any returned pages behind in the next attempt. */ if (cc->nr_freepages > 0) { - unsigned long free_pfn = release_freepages(&cc->freepages); + unsigned long free_pfn = release_free_list(cc->freepages); cc->nr_freepages = 0; VM_BUG_ON(free_pfn == 0); @@ -2653,7 +2698,6 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) trace_mm_compaction_end(cc, start_pfn, end_pfn, sync, ret); - VM_BUG_ON(!list_empty(&cc->freepages)); VM_BUG_ON(!list_empty(&cc->migratepages)); return ret; diff --git a/mm/internal.h b/mm/internal.h index 8450562744cf..46c8bb94ecbb 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -473,6 +473,11 @@ int split_free_page(struct page *free_page, /* * in mm/compaction.c */ + +struct page_list { + struct list_head pages; + unsigned long nr_pages; +}; /* * compact_control is used to track pages being migrated and the free pages * they are being migrated to during memory compaction. The free_pfn starts @@ -481,7 +486,7 @@ int split_free_page(struct page *free_page, * completes when free_pfn <= migrate_pfn */ struct compact_control { - struct list_head freepages; /* List of free pages to migrate to */ + struct page_list freepages[MAX_ORDER + 1]; /* List of free pages to migrate to */ struct list_head migratepages; /* List of pages being migrated */ unsigned int nr_freepages; /* Number of isolated free pages */ unsigned int nr_migratepages; /* Number of pages to migrate */ From patchwork Mon Nov 13 17:01:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13454190 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04F63C4167B for ; Mon, 13 Nov 2023 17:02:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 73C428D0001; Mon, 13 Nov 2023 12:02:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6B1498D0031; Mon, 13 Nov 2023 12:02:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 481528D0001; Mon, 13 Nov 2023 12:02:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 28E748D002E for ; Mon, 13 Nov 2023 12:02:29 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 00552C0910 for ; Mon, 13 Nov 2023 17:02:28 +0000 (UTC) X-FDA: 81453549576.13.1249B94 Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) by imf24.hostedemail.com (Postfix) with ESMTP id B8A5C180064 for ; Mon, 13 Nov 2023 17:02:22 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=sent.com header.s=fm3 header.b=xsBipDat; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=fWrybCjv; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf24.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.26 as permitted sender) smtp.mailfrom=zi.yan@sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1699894942; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NjTQdoHSfkchZC9ErjtVbGgxIgagQAkQl6Bv0Pdns3o=; b=SP2N9Vyuh2Ybd9blXXuAdLPPvI0PB/yazzJCe5ZOUpy0ZpDyrYjic0NXxK8BcFs6oSI9ku zUXnk1b1PagEMaZiTcU0nGYFxEnc45WJYTWqaGB8uFOCeOlcVVquXV+aes6Fu1Iwa6tgsU 1w6oRLhc4BuVTUUHczbIJQi3KtZ85qo= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=sent.com header.s=fm3 header.b=xsBipDat; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=fWrybCjv; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf24.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.26 as permitted sender) smtp.mailfrom=zi.yan@sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1699894942; a=rsa-sha256; cv=none; b=asXOlmsxEg2R+rfPsABcEuXrUywWZhtpi3YxJSBL/XS9yV4bTPH5oOdrt+NV5Lo41IF5OK 7l3O2GAuZqYyBQ89bEKFOOzXjCBJ987duqbUmmq7vnfe55NAwQgD6msL5/7Okby9V+HELk /nuvn2TbVVLDlQ5QIE5s4ZDJ30PZtQo= Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id EC8375C01C0; Mon, 13 Nov 2023 12:02:21 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 13 Nov 2023 12:02:21 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:sender:subject:subject:to:to; s= fm3; t=1699894941; x=1699981341; bh=NjTQdoHSfkchZC9ErjtVbGgxIgag QAkQl6Bv0Pdns3o=; b=xsBipDatk1P2U27rv1kUFkwMhX791z5A0ZKrm9KsdRqC gK6LlO03b005sR4uhJ4VXn3/9VPBjEHQAMKvUVOv7m97BXF8z5zMtwC3ek/ARRhF R6v4qyrbVM0PFppvmB6I0q3RMcI3XpCo9wukEu2BBKS5uK7l4ihGaoOTmEPr3uxr svRq7x+dKmgUJLdrqLGNvLQlhVBUWAmpTSNILEd4R2eguie/IUl5kxGip9n8nK7h VDYs+w5hlskJ1Vz1joC4tprZsQdQ0bJwEblbA1M8i/4tVtw7ss08GvSGirrCjwbX oUYKngh5vZP3UEq3PJZRcMkOejKm03tUkeGpA5Z10A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1699894941; x=1699981341; bh=NjTQdoHSfkchZC9ErjtVbGgxIgag QAkQl6Bv0Pdns3o=; b=fWrybCjv/T/lN8stCivDytBqdHgfEhrf4OaHsDT1O5FD 4WUOTiYtFuR1PHaSD6uYBAaSfGOmJtckDM4l3a8ngGYYKTuyQPHtPHNeRpctctNU CIoksa10PwZKw4Y18dINEDXQc9Iu1uku41mhRD8qc8lhyB0DdUgdt2lSwMBjQD59 lkTzkCpwoAx77imEhu1tP+vmNmSAgFEPIh9Dg/aX6csqf5wf9xlQc26XLh0oE+GA UPuz+ENFtAnSdnna61Lu8ieDd3c2x3Ta8aEC1kHhBgYlSkRzFq+7jM7fPjBwG8/y FBcBCLXq4rHtb6oOkAVDkQLkrNAXAbrkAc1Kpih2LQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrudeftddgleefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhhrgggtgfesthekredtredtjeenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepje ekteekffelleekudfftdefvddtjeejuedtuedtteegjefgvedtfedujeekieevnecuvehl uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 13 Nov 2023 12:02:20 -0500 (EST) From: Zi Yan To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Zi Yan , "Huang, Ying" , Ryan Roberts , Andrew Morton , "Matthew Wilcox (Oracle)" , David Hildenbrand , "Yin, Fengwei" , Yu Zhao , Vlastimil Babka , "Kirill A . Shutemov" , Johannes Weiner , Baolin Wang , Kemeng Shi , Mel Gorman , Rohan Puri , Mcgrof Chamberlain , Adam Manzanares , "Vishal Moola (Oracle)" Subject: [PATCH v1 3/4] mm/compaction: optimize >0 order folio compaction with free page split. Date: Mon, 13 Nov 2023 12:01:56 -0500 Message-ID: <20231113170157.280181-4-zi.yan@sent.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231113170157.280181-1-zi.yan@sent.com> References: <20231113170157.280181-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspamd-Queue-Id: B8A5C180064 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: pbgiedwxj8cxx5eu96hjouak4uju3sxt X-HE-Tag: 1699894942-412613 X-HE-Meta: U2FsdGVkX1/qdsGWrrf8TsZnWOQHpyrA5vPSeiPyz4EcDv1tufHa2YCzN6opCL20IMnEDp3UGECcgTvMstf0SiIJoXZCLCTcAEpSQIVjxzI76llNEkl3qC99avSyn4Hx9t6JfpgOwCzFOlkjHP2DH2Ci4B5HfhVDYzf22/6vm60dY4FAliXWm3cD18bgjtEPEKIMOlPiFsTUBiRaOUrIPN4Z3m0K6NmI1MST37gR7unehysxcHIEiZ3Zjm0tCEAXjKx8UhCL3csE8a892OFSzsTiEs7MxAhF28nF9SBdO1anafEku5xO3CemLgj0ZRbCBWhNMSpWJ7XFrvQiqPn+9Eb29HM0FODQcFM9HnfGeSA7kTGCbDv9x+j6NYJjWaoH9YBlm8MigJNFXyQB+0gMv4Hpt+r7o/CX4dXGCpoIXzw53r9KtUY8T1X1pNOGVjuAyL9uZzyEclnRmQUNNUxz0xkP9g5WPnFQyivxDZgc3bR1Y0qk4XUsM3JFRJw/EHeLW4gTajJFnyigAysDwu+vRnuVFHMuLVjhHYep4rh+HDL5ioDoYGvcrXYtbK7qOsj2SL3BOnYWEqqwgh/5gWA29FesFFcM6G9NK5sSP/FteZJUxTHWq7LWcTSQXqyFQwMxamXOZG3qhVVTfqSOU1BpbFq/PLGGSAhCRkMD48FyMOOfEFXyKcVTd8Jp7s5BkPJHtizM0E1ct+jpVxYnEsev54IH7+9SjljnqSgcDKBRyjMvH+iAZM8yhFBALiwkpvIjMe7gpMqeaZrnJ1YPOF7JRtDc0CF261ILtmdQ6XIUmG5g/X919dKcl9aYQWjxwMEXr2rbU4c/MsqL0Ti+wu1YlSUaHE9b3l0hbv1Tt4+C79ib/chnRPcsnPm7TaOA2g7/3bgTzkBOnPI80r7Uwuh/RvOM53rUy2kCarsS4glH/fIyxgGQfmg4jOso7os1lT6HiDhf0RVQ3nHB8V5kjy4 bH1l1U9B c2vtQpmvDl3CRor/dv4urNEU+zs8eB/L/KgsQjezacQEupOprDXa5g3/UeeJka6KmwaOQkzFOEGAHeWnP3FjUR8/aJEkpxOy99ayheo9V1SWclxDoIdK8UwuecJHwvPNPN3hryJPQRxcjgTF52cMaXvBX7S5m2tC6Hu8f91R7PP+i//60/cwbw5dgotVkng2vMf0XOHdoTG+/fs4M+OZQcaQ1Y9HtLUxKxUh/6iaTT8W65OFkJSn44o1S5WHsdl4Pxp0EWX9Ln0/6qVhOVdGh20RQ08epACYXC6ArUrxbiinigrYuNEfwvhPKRkbdPY1bjAHaSgW4TALIhhYbaaWblWw1PjkeqlyzXo0eRXJ/col3P2h1aWioPrXTvlBej5+Lk25TEG9zHNlsJ0xv7Z9TYdIRBK3yFlqW+jGP783UdpvMyvmWyyzMcus43c8qU7eA5bI0ikD2zkcfLoYD1uNNxhJS3kuMsyXK19hMKEADu+u5Axc660JYPe5CwKqjQAaIuwUHsM7AE+5M7IY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zi Yan During migration in a memory compaction, free pages are placed in an array of page lists based on their order. But the desired free page order (i.e., the order of a source page) might not be always present, thus leading to migration failures. Split a high order free pages when source migration page has a lower order to increase migration successful rate. Note: merging free pages when a migration fails and a lower order free page is returned via compaction_free() is possible, but there is too much work. Since the free pages are not buddy pages, it is hard to identify these free pages using existing PFN-based page merging algorithm. Signed-off-by: Zi Yan --- mm/compaction.c | 40 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/mm/compaction.c b/mm/compaction.c index ec6b5cc7e907..9c083e6b399a 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1806,9 +1806,46 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data) struct compact_control *cc = (struct compact_control *)data; struct folio *dst; int order = folio_order(src); + bool has_isolated_pages = false; +again: if (!cc->freepages[order].nr_pages) { - isolate_freepages(cc); + int i; + + for (i = order + 1; i <= MAX_ORDER; i++) { + if (cc->freepages[i].nr_pages) { + struct page *freepage = + list_first_entry(&cc->freepages[i].pages, + struct page, lru); + + int start_order = i; + unsigned long size = 1 << start_order; + + list_del(&freepage->lru); + cc->freepages[i].nr_pages--; + + while (start_order > order) { + start_order--; + size >>= 1; + + list_add(&freepage[size].lru, + &cc->freepages[start_order].pages); + cc->freepages[start_order].nr_pages++; + set_page_private(&freepage[size], start_order); + } + post_alloc_hook(freepage, order, __GFP_MOVABLE); + if (order) + prep_compound_page(freepage, order); + dst = page_folio(freepage); + goto done; + } + } + if (!has_isolated_pages) { + isolate_freepages(cc); + has_isolated_pages = true; + goto again; + } + if (!cc->freepages[order].nr_pages) return NULL; } @@ -1819,6 +1856,7 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data) post_alloc_hook(&dst->page, order, __GFP_MOVABLE); if (order) prep_compound_page(&dst->page, order); +done: cc->nr_freepages -= 1 << order; return page_rmappable_folio(&dst->page); } From patchwork Mon Nov 13 17:01:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13454189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9FDBC4167D for ; Mon, 13 Nov 2023 17:02:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 548628D0030; Mon, 13 Nov 2023 12:02:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F70A8D002E; Mon, 13 Nov 2023 12:02:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 372E58D0030; Mon, 13 Nov 2023 12:02:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 254BD8D0001 for ; Mon, 13 Nov 2023 12:02:29 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 03F6940600 for ; Mon, 13 Nov 2023 17:02:28 +0000 (UTC) X-FDA: 81453549618.18.A76750A Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) by imf13.hostedemail.com (Postfix) with ESMTP id 43E922004B for ; Mon, 13 Nov 2023 17:02:25 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=sent.com header.s=fm3 header.b=LLVv3ylo; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=BY18zVII; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf13.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.26 as permitted sender) smtp.mailfrom=zi.yan@sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1699894945; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nzOUpjfyn/JJJn/Enob781hONORF3mJcpBIoO1y6HOc=; b=CO0el7r403+JvHIH9qxQPhZZs56EbvvbXM999Kp+U2M9rXpgy7z9FrgbOPgqIGGhVQANMD I+AJ8yVCy1HzY3jCxJ0DZIllp9iClLrZc3NoJiNK8yRX491lWRIQQMYXPEoMQywnvNmgwc IwZkK4lopDErTbcz4Il7i5AzZ2diTQo= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=sent.com header.s=fm3 header.b=LLVv3ylo; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=BY18zVII; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf13.hostedemail.com: domain of zi.yan@sent.com designates 66.111.4.26 as permitted sender) smtp.mailfrom=zi.yan@sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1699894945; a=rsa-sha256; cv=none; b=i1fIM96MfurqRbn46OkRdlqE40vvI5d4lXlEDRCtcvAwlbAGsO5TK+9RIlNUgPppOZcG4s +7EHeSFgbisE32mIZ0bWwwgSojbi6pEa/jKWjlX8yzAIsjDGcz3J7AvKfd6gaovvOEaCnm JcACw4wqU0f1d33mYNfGLvJXRGfrUU0= Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 829795C0099; Mon, 13 Nov 2023 12:02:24 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Mon, 13 Nov 2023 12:02:24 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:sender:subject:subject:to:to; s= fm3; t=1699894944; x=1699981344; bh=nzOUpjfyn/JJJn/Enob781hONORF 3mJcpBIoO1y6HOc=; b=LLVv3yloNFUmMZ6xGakYy4VMI5jabZP1oX9qTx7H16gO 1CZG/Vu4y682pbGiU601G2tvZkKoGS/cQelvSjNl8IyVRCvBxY6ZDp/I8Z0El/7X 9ZfcY15EYusoFRVwvMbdZP6TQvbCHhLECsmgGAFbJf9HFgYb01aD31ORb3xofy3Y 7MIOzM0o59YHJxOvwcxZHA0/sbuL5Je60bq6A/+EhOnwDSY4kDNInc0UL2GwUU2y KoMrFOjoHe/NngSo17AreCt7IaANttkYOr1/Q8o2fk4fb2YI47vmTdRL2ARxdrcs 37aplOkVJOt1jGNxORVOB3T9vRyR7rMhKXKxozk+5g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1699894944; x=1699981344; bh=nzOUpjfyn/JJJn/Enob781hONORF 3mJcpBIoO1y6HOc=; b=BY18zVIIhn/w0FEXwxzpd9RefEDeDi6eMLhNRbpMZAVb 9dGnvGsCq+Z6W22Gby2w0+eOoExdovlBnvJ84qpX5Plpj10dvXTreo4EGwYu7EXN C0G1KpE+tNFcW/GqNrybEqajzff+3jNMkIQdJaYehnaeJl2Dwkc2E0poBSk45Qbk nCPaOWGnMh/SBtyKU3va+yhTH7DvRNKe9NfqaeaXfGUbEQDjCaGg6XRfUbGaQ1YU 3hnkKo6HsZhFLBGIbuxZHZCximCfxM2GOv+SU6jOJyn/jEb74S9EN0ECVc+h43Bf xVLHDT/QAI2Qqt1sAZQ5wYKxhrWruO0RmuXr7juCMQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrudeftddgleefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhhrgggtgfesthekredtredtjeenucfhrhhomhepkghi ucgjrghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepje ekteekffelleekudfftdefvddtjeejuedtuedtteegjefgvedtfedujeekieevnecuvehl uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnse hsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 13 Nov 2023 12:02:22 -0500 (EST) From: Zi Yan To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Zi Yan , "Huang, Ying" , Ryan Roberts , Andrew Morton , "Matthew Wilcox (Oracle)" , David Hildenbrand , "Yin, Fengwei" , Yu Zhao , Vlastimil Babka , "Kirill A . Shutemov" , Johannes Weiner , Baolin Wang , Kemeng Shi , Mel Gorman , Rohan Puri , Mcgrof Chamberlain , Adam Manzanares , "Vishal Moola (Oracle)" Subject: [PATCH v1 4/4] mm/compaction: optimize >0 order folio compaction by sorting source pages. Date: Mon, 13 Nov 2023 12:01:57 -0500 Message-ID: <20231113170157.280181-5-zi.yan@sent.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231113170157.280181-1-zi.yan@sent.com> References: <20231113170157.280181-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 43E922004B X-Stat-Signature: 7ymcqkk5fdkupq6t8i38u76xobh5x4g1 X-HE-Tag: 1699894945-829744 X-HE-Meta: U2FsdGVkX19DV7JV7pkRSg19cowLPSqohXLrVtef0lev0aTB0awy8kqIbH2wZ88/LC0+gc+otZQTePKz6+TQaTYqliUpwu1xmttq8mRRt6FV2KfBrK7y/KCuTa7pYvfCDBUi8Ew8U5IZo2oQUxE82QGbiBiFxvcPohQBP7WJq1/W00dvDWy9pT97lkF5cYIY1Xl+sF3v0GqX/PRj08SO2cUGDFXHm93biOJn8bwO4arls3AEgPb3aw56xvP0ccb36htm40Mryr2KkF5TkofOU8/dSo0UmEnhJ2Mbr2zXV4XjEqGu6QusKFTlHGfszmbbfgCGiHGjcxQkzgZrWldP1KECbGWNSQ365u4TjtR1mL7rQIiXw6+FqNToXKh2vUhtl9dMYh5hdRa645yG003yOfRSHCAAF6NA/6ct9Sq4obpENKsExk3SBsmuCxyBoSMQHvKhMJA3muixTTe2djo/9s2HuVsEdhDmb3xVpa9yzj+Omc1QwVUR6FSge+pS2/yAiKu5hmWqn9DOcO7ELe36DuB2K8AG/JFvM0CAliWqfnRM41pUjl0qvS7uEd2arOqLlenrZdcU5CHEPWMnlm8bgNkY4owB01OsBSt+HafI1UPxS8ROS82JTNrCBMhJxVIH5zkU3v2jKYhbts5hu8182W/y7iIs4Tn/+RIS7YE/ZpGSe7AC4UoFrARkSyYs/48iSbLIXB5i/m+JUtj+CijSp0GYavVxHxVlfTClvMTwEw8PnHH3um8mQ6pN68X3Z26/BhIO8h9rgY9CJvkzqqHi8a0bUiKIEFHoClsKfua8rJ519ppDnocJKqwLq8CXD3HZ4xOCEJ5UItT6MuFZhjQoDdgHUjwdK+644tmCBaGHbXob4tJV0pK1jZjXPoJRTBSSH7bSCcDRpMXTC+F+UeSmKR+HUeCA0ItGijipwuGyApEiYQnjPjGJYvM6/1aNvip22nAAaDcwEaFcXhuy+Q9 sBj8HtmR Rb74f2NiM/++X3kAryzMQcuAbLuYYWFY8vrRf1K6dqC7HzoXZ+OOo9OE/RLNWjtZa2nQLjsJYUxo8WcsyzXpdmEeuJD7KzOASlsRZzi4skI8fhZW/NceffpsUXAxozWsRon7r7On60L1SwMFXxyk8Ttksyujb4ZfSvJfQoM4Y0TMG5WxwdxajY7PW+mnTRPBvpaiY1IXPXCv7Jp8TJmsDd182tg9OPTCGiKFUqtAeMHJSGWOxEHpv9UcVehQGTTHX8v2oT0AXX+ni17Ojv2EVu5av7tj0F3kdvpuHxcutyRBhRojoM82cv3+xyfVogEIuSdEm/XdDBMm9PIsEZV0rKLda2nLjYWPq0zC9LrFAmkBcnRWt5tBnmA6Vs+WkqQgfg4HUUIBAoZRIo70AkCYX5TB46fg2Dym/M5bnPsY/mX6FfLM8BqbcII4X61hn5B5KPtlVZTT0P4zHG1Ch9NFDfjCgSefSEaeTENogl6Zz3oCKqmUGsVOh2g212vQT5QnwyeU3K7E1DFvTpd29re4RTXIFAU4/Wlhbh4IQwiEhNHKR/lXiebm4ilJ7Vg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zi Yan It should maximize high order free page use and minimize free page splits. It might be useful before free page merging is implemented. Signed-off-by: Zi Yan --- mm/compaction.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/mm/compaction.c b/mm/compaction.c index 9c083e6b399a..91809bee5422 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -127,6 +127,37 @@ static unsigned long release_free_list(struct page_list *freepages) } #ifdef CONFIG_COMPACTION + +static void sort_folios_by_order(struct list_head *pages) +{ + struct page_list page_list[MAX_ORDER + 1]; + int order; + struct folio *folio, *next; + + for (order = 0; order <= MAX_ORDER; order++) + init_page_list(&page_list[order]); + + list_for_each_entry_safe(folio, next, pages, lru) { + order = folio_order(folio); + + if (order > MAX_ORDER) + continue; + + list_move(&folio->lru, &page_list[order].pages); + page_list[order].nr_pages++; + } + + for (order = MAX_ORDER; order >= 0; order--) { + if (page_list[order].nr_pages) { + + list_for_each_entry_safe(folio, next, + &page_list[order].pages, lru) { + list_move_tail(&folio->lru, pages); + } + } + } +} + bool PageMovable(struct page *page) { const struct movable_operations *mops; @@ -2639,6 +2670,8 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) pageblock_start_pfn(cc->migrate_pfn - 1)); } + sort_folios_by_order(&cc->migratepages); + err = migrate_pages(&cc->migratepages, compaction_alloc, compaction_free, (unsigned long)cc, cc->mode, MR_COMPACTION, &nr_succeeded);