From patchwork Thu Dec 5 00:18:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13894609 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DB16E7716F for ; Thu, 5 Dec 2024 00:21:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 131186B009F; Wed, 4 Dec 2024 19:21:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0E1826B00A0; Wed, 4 Dec 2024 19:21:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E9D866B00A1; Wed, 4 Dec 2024 19:21:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C667F6B009F for ; Wed, 4 Dec 2024 19:21:06 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 88C9AAED0E for ; Thu, 5 Dec 2024 00:21:06 +0000 (UTC) X-FDA: 82858999818.19.8321EB1 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2051.outbound.protection.outlook.com [40.107.243.51]) by imf23.hostedemail.com (Postfix) with ESMTP id B787E140004 for ; Thu, 5 Dec 2024 00:20:53 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=ZUrlaFgw; spf=pass (imf23.hostedemail.com: domain of ziy@nvidia.com designates 40.107.243.51 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733358053; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aecB1bzSv7lUIxU7S6qbIrcsPh+7bGet1pzIbTFfV5s=; b=NC1fZyRvjOlM/Q/IbFjU7m3ywDqQ2CdhFt71kZSCESuXMRH33OCZLSTtAz91bdIjhVpg1R EBRyjO3wF233WUmely61VsraLVC7JYcpKpt7YnbZDbXtLsqm4PAMXkbfH9xtbRB/j1378D aPmJ/SLQ+gvLzh4AkymQoB7EbuSMiIE= ARC-Authentication-Results: i=2; imf23.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=ZUrlaFgw; spf=pass (imf23.hostedemail.com: domain of ziy@nvidia.com designates 40.107.243.51 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1733358053; a=rsa-sha256; cv=pass; b=fRja1IiyDu8szvtk4wRmOFmKTN9ZPmUOdzn4hk8sDtx77+Z6jOCGr2hhNIxAW48Yt3cSlp NxyGT3Na846QYK+V9hb4QpcX0rGGk/wEpGwY3eUnI9ry9oyGqEhIvEZ5TYfnmuzzEas693 Yc/Ee2u8S2DJK/2SzT9yekYQQM84UnY= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=A6jE9GEgRd+teJ/e5Dcuio5DT+TsLXpnJ5UI8Jw3EWgE0HMHz9zlVNjGtsSqNmvO67RkkIlOzl8UQVbZr58WSByEyBtpMnf7bxIlgpNFJJJP9AUtP06p2JHCzFPJUujD7iy7pGnk/IajMf8GBIbjedQoaquPEUT49RUaoO4URNyQeNsU5NPHqFeIvhwwwhviG1NxbRLRp3H1SP97n8MZy9BLivpvucj0NxA91OkF+CHVRBucCJCis74hZuhHXzD1LByA5K4NVIPTeYaNNzaIRT4SOI2Pvd/K5zjj7E1mHFIQOoYOpVXQzIX3VkmCBb251N9JugoSq7mKfd+eqU9p0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=aecB1bzSv7lUIxU7S6qbIrcsPh+7bGet1pzIbTFfV5s=; b=lEdtsd5i+MatIalLwsN7qLi5PhRnvn3CsmmR34c1RnsSeShv5fDpOROiPDGXTpQ+VKn9yF7YJwDT6gaJxvFs/AhdQ2uQn/Wc7OcBT5j2thNvRlMXv08fX/glQ1ovmtANa4U02B99OzhiYuoK19UyRne4ioxEokOp4U6s2FlcPWSrlP0wAM9wgNlFAkgE5wiEj+PKhoMpdDyVsRGvJbQqgvpzqpR3c7FweLBLLdHJ17zyqmeIdcc8M07EP4b/y/xeXgJsV8G2shkYqCIe+WXf0+u8VIN1jcEgpcTt4coYJigvP/cn9F/RXZ5WoLmAa+ts8e4a29XtC6JpfaasdaDgkw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aecB1bzSv7lUIxU7S6qbIrcsPh+7bGet1pzIbTFfV5s=; b=ZUrlaFgwaKRBFg48x+1ITBRlj0TUjkgbDOFooR194CoKjTOWpB764aYlJscZ6/NGWnpdWHW94VChbDMKquWfqJlT5rNWW6O1z+XoB2Sgu8kU8dF4Ob8hfeBEp8g7CQwMQKB+psVKGh168R3OI7YN5+3bRZTqRoye8TITqw2nsqKsXCM4SyARRycQKtHCPzLxxo13OiSea3t2xaN3q/YV5ptkny1V7c/TFrjiEdYWJj1XeSkywJ4s5z1PbhmBjr8213YmveKOZ+eFPIDwGLwEsBmpYNaMW9YSWVscZlbZveTK93B3QFIld9APk9XItwjcpyp6za53rtFI70qMid9pWg== Received: from BL4PR12MB9478.namprd12.prod.outlook.com (2603:10b6:208:58e::9) by IA1PR12MB7589.namprd12.prod.outlook.com (2603:10b6:208:42b::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8230.12; Thu, 5 Dec 2024 00:18:47 +0000 Received: from BL4PR12MB9478.namprd12.prod.outlook.com ([fe80::b90:212f:996:6eb9]) by BL4PR12MB9478.namprd12.prod.outlook.com ([fe80::b90:212f:996:6eb9%6]) with mapi id 15.20.8230.010; Thu, 5 Dec 2024 00:18:47 +0000 From: Zi Yan To: linux-mm@kvack.org, "Kirill A . Shutemov" , "Matthew Wilcox (Oracle)" Cc: Ryan Roberts , Hugh Dickins , David Hildenbrand , Yang Shi , Miaohe Lin , Kefeng Wang , Yu Zhao , John Hubbard , linux-kernel@vger.kernel.org, Zi Yan Subject: [PATCH RESEND v3 3/9] mm/huge_memory: add buddy allocator like folio_split() Date: Wed, 4 Dec 2024 19:18:33 -0500 Message-ID: <20241205001839.2582020-4-ziy@nvidia.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241205001839.2582020-1-ziy@nvidia.com> References: <20241205001839.2582020-1-ziy@nvidia.com> X-ClientProxiedBy: BN0PR04CA0084.namprd04.prod.outlook.com (2603:10b6:408:ea::29) To BL4PR12MB9478.namprd12.prod.outlook.com (2603:10b6:208:58e::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL4PR12MB9478:EE_|IA1PR12MB7589:EE_ X-MS-Office365-Filtering-Correlation-Id: e77bfd14-1ad6-4963-57a7-08dd14c262be X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: Z1fvhnqYpg4VtTjRMkmHYsA9KR0tVqTf0HA1/FKxnAzfVZZsp7xQSYLfy84k/wS6CnQeAgpe8GWy7X5ierCuV2hrwvUZYr53MvXBAoqfKHypm1uFzDYvH7cig8Ue7yqBflM2lF0cnN/OyQgLFa8Xx/KnBBq5Kw2Kxozjy6uzY5f+FHGchoWCxo0Vy/RnF5BrdaqGZK7d4sOY96SU+QAn6m33yUbtzZbfx6UwuY1am6NzSiUQhoRlTR3zWyPutEo6epqsiGUfE9BJY+hc+LZ45YsODrZfXolSt+VKIvOoEXhpnl8Y3w2vHFevN3dWU+7DBd2Ep//zT/Qz2mrU1F7jLoS7sj3Mz0Fipeuwg3aUYfP6Jo0zZ1C7rkBJlWDxHFa9GoTmbxNfEniEz1yZw9kN9TGvsiCi0lxpDxJRsxUJM64DY/59Uh+4zPirvFz44Ahg4d3OVuBN6mZjJmrpZXyV8Deqz+9ZpszaXc5l6BzphHBUHgIGtiEWpJLucA8kwTNC4OvGqyk+WHYdfCgjoeJgg2GvNBs0vnP7WAIouu5OTvezWvuenmyAylU3KcR5zosFSR1u6In2FR66kd4LGXMz3stz9drEpaQJQUq9BudvTpJJCWkNaZ6XHm/2fqX3HfcYXTO/9Ablw/bcB0F48Y7CyvDxfD4UqOOQPxe6fnh8o2q0lWXz+s9axieR4D4fcmh5boi1Am3wodZgSXeo6GLyyxga5crfwhyIFtkkCTQ5tNtnOEjg2s67raoQghPxAhyNW7r/2hnQsmOShEtunoBEQrPukHg9f9U3nAIxKOchPnSeTu8npoVa/uFMyxLtbisP2t3xGeYRXRsnf9ZZbfYFq1U4iUW9KbJJ1y8sH011wVWgSvkKFHEWK8b6VwrP5tzZGrtIGgXTUdnzbs0jQCGD0pPFrQlJvgJnsnJrAKPdTct6BpLJuhez8Cc72UMMg85hse+T1RQsdljnY2AjRLNefTelY/GADOGmC6yf4nLm76lJCclpuXWMVAjUMuyWWg8EqJMItMpKm/KrNSReOpAdFX9kYIionoAAzT2/HxUSx2jDoCi5yeUkqtyM9Fv3gfUwjlAtP9wtDSvBf1PvsPSRCD5RqimNbPCRQPbR1bm2EHFvAuHxwByAF9HsQ7K4OE0jqS6WD56c9kGWgV+latAiqLQC7UPeDYmp7RkUGO4wnkhWDUFrE8L1xMJqWtZ5tr5E2Wsd/ryVWRmDCPFXfTJCS/bTkdJ72Yjg6mnwsdNMhfcFo+Wt/3SwBNO3YWbzVogok0bufP/XlYEkVA6fCN7bALg6H6HHe+8XkgH8gvd1eujibk2s8eOcIKI7F208PR3GIjEugcBEnllNVZT2aKUK3g== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BL4PR12MB9478.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: hNDNGPSm8DlF8Li1QB4WiuJ5v0uu++9w0jo+TPELyeHwumBs8WfdkfFSiin9/er4bzpKXVdA67VJsil6QBZo/ox7jiwLScfpM6gA6k8j9uqW78T6Vpj+iuALvH+WR/9ypj26UhuNMDbVW75RVHCu02u77Am9JzKNfKbkk0IWqvYp4Xz+dMuHyv65UW195vxsfoedN5OZ7bvbL33Up91vmscP9MJSzrKDzmjGZ3ImgdDYUh9QrZWDOFzK4+6OE0sBTPxNKRTk+L7a1k1bJw93DsrQ7A2IEc/GTjyeacZzrhJb1DyFWbYBocqDFCT1r7+wtRsGgvbmnH6PgIJxl5fADQd4hmwkYsexunIt0aWj8MRReakepBw3ClsRD/PzmmCCqih7elhZkiV6NKBJeLMkfZ+AHywcZ98LppjgWCFGkQc4cWI4BZpFYFPG1vHTxS0BJqF7Jgol9tgdpw8i/HC6vTigtPHca00xA6DE8ybdYBRh42DAVgFHbIZHLH5sQ2BMnAyeiCFjkOYZ0IcDJwlZgLVux46ym/g92YCekyUntDida2Tm2wlSES7U6Te0hdboTbk6s/Dix5yg4jbH8qVtoQDv90kJVJNSXIZqLHu2ojNtfzZLj+aTOEko/zH0UPPC0KzpObzirdJKXElyB/NbMm0y3RT6C9ynlO28aDj/KzCbbx/7CoKioQn03wkOXVPKDZKcjvJK1IBdg8b8pUuz4+CkGAxMT7CiJWbnEivG3wLh5VZOnGDbWZ9ycUq95pWoEotlti+mU3H0/aQCuNbDdvj/984Khk2o9lNaz1iMQnXyS59ZEzyVUmomf/8GhFx46Cj0ut6Lk/HleC9RE5tWMk76mD8+Ov3oR7+LM/fNUjyzkXVnYWDzyr5XltdBs48z28DHxlPycOUOyYxcfTpF+nF2BqKW4lhKz9QmDlgjE5itZ5UrgpJKxWvilIj1xN1VHzDvHvw6RDQE0FEdvylkrmb1CR9mwz3/Ukhy3hqo7Dxjf8ssyIieqFUALSW5yWCKQYAgrJe0DlahqjYwVxBgdndMxKv+Mgbo2kvSquR99WMfUlm+RxxFfXJLn3M2qgmQFvhLF8zy2DP6XTEX5Xxq2C16IsqCzS7vBX+cvocR4NvJkWbDFf194KaBQWAwoIjrOL3yyKjGqvW6Bo4/V3E7MG/q/Q78fNyzk7FYAs1/FoclLpziMt52hLwWzKR8j8qD/y54jDQy6LXLCIKsENF5sf+Jd0KrB+/hlcr9g8h3L0MSzGJ2bhCOnezqQzF6koKxUv2l0NN4T8LQhlGy4EyDD+IzNEG5+J860XrHG2uq9hw4a6aLspGU99lwIiL7Nok87WaTolW1W7PxhZFr+ctikic4Dm8lTxMGo2xICvqlN2vU+p1PpxfxftrzKmEhG7VgutGMeuhAAErFauRRVSygEZgld1SsdI57soM+C6LFohIZc7m22iiSDwug6xxXqEYjhu+E1WR+K1TO0tM1BucIL2d9uZyvxVWQtX9moZXTw8AJiX22afGvRPyryjb8UYKoKyqmqhM1dvzFttWiXk0qGLcXf2QyoP80jM5GBfq4zCf/NGZkZbiC6zBc7vC+tI36 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: e77bfd14-1ad6-4963-57a7-08dd14c262be X-MS-Exchange-CrossTenant-AuthSource: BL4PR12MB9478.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Dec 2024 00:18:47.0320 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: UHrPxPob2I8r/EODqtdVkCgs1zISpQHa+QwglnI0OSdQH1VdK2eRH6lFovU9tV7s X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB7589 X-Rspamd-Server: rspam05 X-Stat-Signature: 3ob1oy76ryk1wr6fai4med7xhjr5n3yu X-Rspamd-Queue-Id: B787E140004 X-Rspam-User: X-HE-Tag: 1733358053-908897 X-HE-Meta: U2FsdGVkX18OjCozfYyChNHAbpOg9c2GhaSTF4MWmoEyauS/ZJOVu6TDXTUzTMxY3VlQJ+ejlncp1j8d5MXGt9sitRwAfgyAvXxhAQ7/PcGnX/rzsv1aWY0WL/Ko7f2qbPk48lvKq66I/RHGqlQqWyQEq4QGxsWnJBGrCPxTGHqRlf14EoVnp/g9NfeV7MODQZJjd1p9DDcY0ThGpfySsbW7+NrP7CDDggR7kSKzW+yGeAsegFa5jzS3Yfvp1dkXPby/HB378Y/VK42FMllFPUYuYBe+hfNDmfvne2X8PZiUzCjgbAVaKnl41JjmPuO/95FdS2kYBr5B/tgthiBjDhuSmhcQun1YJtb2zzotWFhDKNsVBqTKGd9/hxvmsVvegVqKevRViSm7OzfLsQJi0III39KqR7og6il63lz1D7UO3XihWdYMmFU5ig2cxa/iNGvH7Y1O5bTQDHOhkjrZ9iOrcFz8xZzF1LF/lq7cOW28sSpSpq+EuOgcWmyatQaRtwUFe1BSwQcbNepn9oCWgvaWSYldS+Ox755ql7ZJcrGgqaQTZ12QKBhrd2zZna/EOP8pnRK9PErBYHGWdsKb4DVR999ZOPdhuMdnsghkManA6F8uevPp9OGORXBhCK6dIYFxZ4jg2lk7vjy7QYvuzfqxZVXMj62tpCdwdD3MhVbh931AqDuzJVlhvlURpMSghYrQpQWwRvzfkGNRXry3KQyc3CNFmWrtSvmdxy60e1fdor6g0Lnb/v9VnKnTjoegmkL3noP6m4mJT+l1CtoXaRf1LOd9HmM8jWi83W8Hq8aqoylcDqbxwvmnXYZcr23uKdufBEJx/ptZtUsnfBauAQowoORFxxUwhmnnA+kB4FsKC9wYAFv4DY7YsoDFGP0FufGRZLUaUzR/nE2ZgL8G0hf2G0lBvtpC311cHaUdz9S05/vBefRCVUF/rJ86ONwX81/Ggt/WhymirxYo9ie yIekWVNl hpLJl/e6I62MgQJsxSC/L47K81qpH2Gm9oemO5tSxh7CoSYJPRyivHNSFW6oNqKgz1nsdKFrm5xilkMWLSgZEe6+24ad37Cu9IeGnlnphTCsT5u7UKGXC7A88gOL8EQ9JQICdqoV3AIeKa0p2wpiJWlljG4D1RfeNor/+cGcFvlhiGOjC5DxeZH2z2MkZbiX2R1IJB8UJavzwHChkpOgjnPIIASnwZzOctb+PgnvcC0uV+1YbdYokOCwtiJ9PeDEcIvTqqfoAaW9Uv5wSLOp2Lc5jQupF/bzodudDd1PosVTVBjoJf4hBy+MvnFn/oPEHZGQdOsZVK4vy76aIPGvIAC191BynsRnV7qDuz/ta6QQIBYvkHKxwwVe+enqSlPjm4Bqi6KgbISnG5NE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: folio_split() splits a large folio in the same way as buddy allocator splits a large free page for allocation. The purpose is to minimize the number of folios after the split. For example, if user wants to free the 3rd subpage in a order-9 folio, folio_split() will split the order-9 folio as: O-0, O-0, O-0, O-0, O-2, O-3, O-4, O-5, O-6, O-7, O-8 if it is anon O-1, O-0, O-0, O-2, O-3, O-4, O-5, O-6, O-7, O-9 if it is pagecache Since anon folio does not support order-1 yet. It generates fewer folios than existing page split approach, which splits the order-9 to 512 order-0 folios. folio_split() and existing split_huge_page_to_list_to_order() share the folio unmapping and remapping code in __folio_split() and the common backend split code in __split_unmapped_folio() using uniform_split variable to distinguish their operations. Signed-off-by: Zi Yan --- mm/huge_memory.c | 56 +++++++++++++++++++++++++++++++++++------------- 1 file changed, 41 insertions(+), 15 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e928082be3b2..4f4f3b50c537 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3731,11 +3731,10 @@ static int __split_unmapped_folio(struct folio *folio, int new_order, } static int __folio_split(struct folio *folio, unsigned int new_order, - struct page *page, struct list_head *list) + struct page *page, struct list_head *list, bool uniform_split) { struct deferred_split *ds_queue = get_deferred_split_queue(folio); - /* reset xarray order to new order after split */ - XA_STATE_ORDER(xas, &folio->mapping->i_pages, folio->index, new_order); + XA_STATE(xas, &folio->mapping->i_pages, folio->index); bool is_anon = folio_test_anon(folio); struct address_space *mapping = NULL; struct anon_vma *anon_vma = NULL; @@ -3756,9 +3755,10 @@ static int __folio_split(struct folio *folio, unsigned int new_order, VM_WARN_ONCE(1, "Cannot split to order-1 folio"); return -EINVAL; } - } else if (new_order) { + } else { /* Split shmem folio to non-zero order not supported */ - if (shmem_mapping(folio->mapping)) { + if ((!uniform_split || new_order) && + shmem_mapping(folio->mapping)) { VM_WARN_ONCE(1, "Cannot split shmem folio to non-0 order"); return -EINVAL; @@ -3769,7 +3769,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order, * CONFIG_READ_ONLY_THP_FOR_FS. But in that case, the mapping * does not actually support large folios properly. */ - if (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && + if (new_order && IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && !mapping_large_folio_support(folio->mapping)) { VM_WARN_ONCE(1, "Cannot split file folio to non-0 order"); @@ -3778,7 +3778,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order, } /* Only swapping a whole PMD-mapped folio is supported */ - if (folio_test_swapcache(folio) && new_order) + if (folio_test_swapcache(folio) && (!uniform_split || new_order)) return -EINVAL; is_hzp = is_huge_zero_folio(folio); @@ -3835,10 +3835,13 @@ static int __folio_split(struct folio *folio, unsigned int new_order, goto out; } - xas_split_alloc(&xas, folio, folio_order(folio), gfp); - if (xas_error(&xas)) { - ret = xas_error(&xas); - goto out; + if (uniform_split) { + xas_set_order(&xas, folio->index, new_order); + xas_split_alloc(&xas, folio, folio_order(folio), gfp); + if (xas_error(&xas)) { + ret = xas_error(&xas); + goto out; + } } anon_vma = NULL; @@ -3903,7 +3906,6 @@ static int __folio_split(struct folio *folio, unsigned int new_order, if (mapping) { int nr = folio_nr_pages(folio); - xas_split(&xas, folio, folio_order(folio)); if (folio_test_pmd_mappable(folio) && new_order < HPAGE_PMD_ORDER) { if (folio_test_swapbacked(folio)) { @@ -3921,8 +3923,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order, mod_mthp_stat(order, MTHP_STAT_NR_ANON, -1); mod_mthp_stat(new_order, MTHP_STAT_NR_ANON, 1 << (order - new_order)); } - __split_huge_page(page, list, end, new_order); - ret = 0; + ret = __split_unmapped_folio(page_folio(page), new_order, + page, list, end, &xas, mapping, uniform_split); } else { spin_unlock(&ds_queue->split_queue_lock); fail: @@ -4000,7 +4002,31 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, { struct folio *folio = page_folio(page); - return __folio_split(folio, new_order, page, list); + return __folio_split(folio, new_order, page, list, true); +} + +/* + * folio_split: split a folio at offset_in_new_order to a new_order folio + * @folio: folio to split + * @new_order: the order of the new folio + * @page: a page within the new folio + * + * return: 0: successful, <0 failed (if -ENOMEM is returned, @folio might be + * split but not to @new_order, the caller needs to check) + * + * Split a folio at offset_in_new_order to a new_order folio, leave the + * remaining subpages of the original folio as large as possible. For example, + * split an order-9 folio at its third order-3 subpages to an order-3 folio. + * There are 2^6=64 order-3 subpages in an order-9 folio and the result will be + * a set of folios with different order and the new folio is in bracket: + * [order-4, {order-3}, order-3, order-5, order-6, order-7, order-8]. + * + * After split, folio is left locked for caller. + */ +int folio_split(struct folio *folio, unsigned int new_order, + struct page *page, struct list_head *list) +{ + return __folio_split(folio, new_order, page, list, false); } int min_order_for_split(struct folio *folio)