From patchwork Mon Jan 6 16:55:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13927681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A95A9E77188 for ; Mon, 6 Jan 2025 16:55:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 406AF6B0096; Mon, 6 Jan 2025 11:55:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3B65B6B0098; Mon, 6 Jan 2025 11:55:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 230586B0099; Mon, 6 Jan 2025 11:55:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0494E6B0096 for ; Mon, 6 Jan 2025 11:55:58 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5A69F80128 for ; Mon, 6 Jan 2025 16:55:58 +0000 (UTC) X-FDA: 82977629196.10.57C1ECB Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2053.outbound.protection.outlook.com [40.107.223.53]) by imf08.hostedemail.com (Postfix) with ESMTP id 549D8160021 for ; Mon, 6 Jan 2025 16:55:54 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=cdmTojl8; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf08.hostedemail.com: domain of ziy@nvidia.com designates 40.107.223.53 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736182554; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pxyeeMVuxeaK62VlUKF1S6q9qijzmSMRqC58MRDJ0Bc=; b=3evtWuKFu3T8RZNnvBwzr41Q6fxByfMgNaOFFAK/x6MHaJ+YX8+fMJyG45LSLzEJTweaq1 ZzBxOsxuGCWNNRNjh3/Rk6NbtoAHvfwRbo3Cc8+BNNwvYBIPglaoHXhO+K9vr+fbCb0KYs i1pQvnbGUgbF4lH7RAsjOav8NuUEPOg= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1736182554; a=rsa-sha256; cv=pass; b=C1+oTcIJ2rQhD0amxp+IOA3wwowCrhBL0r0kt1VZdTR8pk5jDYNgT+pNDds34UXW8Wiiku aRt9/OVf2aG4dZZgjSYFWLDMXfduyJooBMQU5NNj/D9rIaYxGUXL+DeMsoyzwGcObruIOB ENI7dBjjSlbSDuHjA5Rq0uTxTJybclw= ARC-Authentication-Results: i=2; imf08.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=cdmTojl8; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf08.hostedemail.com: domain of ziy@nvidia.com designates 40.107.223.53 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=symVzF6MvTRzVyVwGTiTh1jU6P6u3S6YZZrW80eZEwPSHNt4r9MJcwuEpV+QxL88i7FCt4REk6eOTf2/X9+J/ZOlp8gjdFGDxYNEMafSBrzGNNHiAeq45kw3kF1imSfH43v7WZSSv7uCCud4bwo80LCvz2Z3OIwwEMh+W7Ec0dYwSl7Ldo56NBYyy1Rp+AQ2Bv9zB3wjD6wPyi/cG3T1E8xEncQ07ACkIMqCCKuZl9x+zlDSnS9Wn/ICYrHEEbwmnS/xSCH2eXVUZoq2Ah9e8w//UObIXHQLGRo9HpaUSHbA0nHB73hcSpBsCIk8hOW9obwACp8Qsw49TQCvB92Q5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pxyeeMVuxeaK62VlUKF1S6q9qijzmSMRqC58MRDJ0Bc=; b=HioV8nOOtwHagZQ6sgrZ8RkYS6F6mmWmnASBnZg4bBHEqH4hzRyZA/8igxFm6VhiwpNUzMDuR2ZDDBcg0SwaQfdmrOzjlVyq1i8XfFl1aDdILOA8ArA25razbj8bx7+S0++3LUNi53hPRAMLUL2RUwbdaID2/rwjJRLTs1HhNcSjmDpMTj9GEZqHfEGMEF7Ws5imCZdku/7mNAaxOXEy5veXnfD6vqZpK9ypcxgLZx4RDmQ/D2Fe41s1jKSbYBRmlZwgXk48R/wLKspQvCed4LsqR6qklkKLBkiPvzTap23DI+RAY8JVHRzo93Jm7y97n78DpCjd5GgYDqL7+KSErw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pxyeeMVuxeaK62VlUKF1S6q9qijzmSMRqC58MRDJ0Bc=; b=cdmTojl8D/HTf7ush50rYQnmXtUAMVsvkHKXgKV/mgSSm+f1jRJPdi052HdZwXDxdyYeMySFrDWvLj+Lbpt/HR/raMCZGgauGt9KWEpV0SN4COsbKFV2kcwOt6JfY6hcyfFO/mUizNxEmDBZK/XcXYJ8TkpwyjBokskBnZwlq2UTOKDn0icvFmDvuRlEnKtFR+ip0PqUHVMv27ASdu8QQ5q+r5siKpH8cBICNZXFb6hpdJHyrse2Rtl8o9Gb1BtVVu2WAKaOlZoQ8k9KoEeUC4hqt/XBUeRguJV61U0Ui/YPlzRJ7mv9Nob63/oeS0Xhr60wVf2cLSqsvt3+CBu7JA== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by PH7PR12MB6634.namprd12.prod.outlook.com (2603:10b6:510:211::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8314.17; Mon, 6 Jan 2025 16:55:28 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%3]) with mapi id 15.20.8314.018; Mon, 6 Jan 2025 16:55:28 +0000 From: Zi Yan To: linux-mm@kvack.org, "Kirill A . Shutemov" , "Matthew Wilcox (Oracle)" Cc: Ryan Roberts , Hugh Dickins , David Hildenbrand , Yang Shi , Miaohe Lin , Kefeng Wang , Yu Zhao , John Hubbard , linux-kernel@vger.kernel.org, Zi Yan Subject: [PATCH v4 06/10] mm/huge_memory: add buddy allocator like folio_split() Date: Mon, 6 Jan 2025 11:55:09 -0500 Message-ID: <20250106165513.104899-7-ziy@nvidia.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250106165513.104899-1-ziy@nvidia.com> References: <20250106165513.104899-1-ziy@nvidia.com> X-ClientProxiedBy: MN2PR08CA0023.namprd08.prod.outlook.com (2603:10b6:208:239::28) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|PH7PR12MB6634:EE_ X-MS-Office365-Filtering-Correlation-Id: ac137967-caca-4d11-74a5-08dd2e72ec98 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|376014|1800799024; X-Microsoft-Antispam-Message-Info: jDkEPpy7SFhqE7F0bhe7Bfd8kGnas5NVII5Zn231WEHQDLjTOcKKmkqehsr/tKRIbjsL2Y9KoTM/RtMBsBhqr0w5Tho85iU/YoaD2wIjfk+zd/bUxqt5ufQcmrJavLqznKKuHy1fxFtM3ua+u5ExrMpETqatQfet23KuWi/I4BilfGIBqMwWY3o595Gju15ru9qFDju6B8TANy/xTArClLAhNfbEaKuHNrr9nmQX+6ff/Jo94X0Wp+2bhfZjBan5KvFCNjiGOOtPBbwRzpBMO4aC6CFS4Bfw/2oRIeg9Xp0ZWAUMXSqIW4Cg4QPYLiEJjdrlBaqPl/iwiISCQqejJHJl77dJ/cehx7RESdw86b3VYss5pqMYN0YtCJ1Nnw0QszC1k81pyio1JJ1odRpExO55mNIPnu1+9Sg/dyWzVb1BrccF6kuwjb8oAsXAEeSdjrGzgz+2bG/0pMCXMDHO6oE9W2le6tKheky+jgGfEI5otiPCFROXOl3RYdk7EyIWvPoFCP0YAeAfvFq0APyd8i5icxKDtBtnvKff+79HMWLhc2dL8e3Ro62Az/QwGsHcahhJG/EDOw4SU64KVOKZdcDgV/3CknQAHHUMD9OwFmpLnra948Z1xVXKRsLKKDw7N4wnrFOFzxu2bWeWuzHUhocTdgMf3eLi5zwtk3sKZJ6g6wfAB3vktphdItM29+Ol3WN7tgbguixpaskVe0aWeSOJ5NA4kVpblxDsxdIPlbJv/+qy744fdHrQVQ3n1E3Qav8Ni3Kxo6TnLhSFDsJ/dDceD5KurLhjyImCYMGhzDjB75hBIpRwlz7UpHhqR1D6d+a82qqtjhkFUbHgfEPN4mfjOdYGogCDhVzbBlekTTpPtbouyoDvTV/d2fSo7aJ4lmwf9oXJO28n2o9snoJ/EbHcFbmLl9Unxk+VXXEu65yPb75U15lzYK2PF6LEwqQluBZpjykfQV9zGfWMJhRfir/sE2T68a8+CXvGA0496rV2gtKVCGnE4ir4L7KJZxIJ3QGOIguNbPfz+ltysM6fF4NxEhqjYkgNJDjTk/9dUGQQXZktk7U3aBt9555OuSXy73+5YLS5B/gwOwPci9y6KhWTxhKVNA8nKF/Ev81NnZMHic6aywSUbtT0K/38Su/p6LP0xZlJp26bwCH4HsFKvBg2RwqXlR8CKZKsmHoA89TxpAxek0itxxSGWHIqde4EiK2KCFFgdfE9Gaxggi63aasVQqv9L/0IaLLMxbxRQUrVt7LhhGqQXDRNRbtjI0v6gHGxdjcGE3/yTfKA7zpeHUaHyXEKIGib3pdWGDfWHTIr9Ik2poQ81cb1dtD3W6EaQOaZN9bFl2fY9CHfPCU33eYnUiOuj1ZRlWNJBVCJ1yLIXoofI7eLBRK6nmOJAp50 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(7416014)(376014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: EOIK+xFreTBiN/nDbwjIipOLlc9fiRXW3oFfaCJ7qw7JjYKhauJ0RVO8QlKw8MOxf/+VzBxfhpAIBvubk094IWnthhRlDibdfGSY2L7CdduwcD+XBkjfDyvgIB6/fuTKqkpZ37Ushh9wj3IkB3cSgNeuITfkVmsTfL71JSYjc8rCh0IL3Maww9ZEVaQNHeW6z9wAOccheN67/jRDcEGrJI5tL8KfJBKyhDlKuRBF2XxKuDIJj1HagH/t4I/nnWpuV0U7zrRGSj4BR2ApA6WtjZZrozQmAzn6pliGLGmZfsZPQVWqQ+5vXdwWCvoiRsBGK3cg4NZKEeihaknOWNRxDUfSOCgdL641crxMQTsoFFli6ljIUvb1zPP640FzzH1FZHNDHtwjGR3A5amcpGH54YnMi308aT4jVswmXQHdKxSUW2nRKVAJH+1ow7B0yZk57E9W2AFNXGImnDlAUCA61X2IS0ju+O3u2hPAeXIlkNN4G34R1PkJ/SkTPDi/VSKaopNvZYa3fc6vls72FbRjFYdvuw797MUjkwhQ8r9W1Eamut018Io8OAcTTPXAL1W2a4HqYMHQIwLJacRXUNqxovdSjIC+FpEUQLLGmar+tAu/K0CAy5ez5omHpzZpycCI/CT7KIJe3OWJlbZvqzSg3iaT2jYUgrZO7fiYOmr7bI9V2nMLVd2/27MGRLSc9eyY5f1gORLbMiuYJqPPs66ZzD67MTu80GLUxbiXrhbJaR0WLG8HLdOqFqhsSDkV61LKnsuKJss0Tv6aUUJZ5H7EitYQvnCEMHfWaJ/spK1cGmMJMdN5+sIKVyNwtueBjZqYBSBDr5VIDyZ/HciVXRbL4oVBsCiyZedaiYTpEBUBQ6MIUtxwRqQguXQofAJYTXZL5i2cJe9y4MbJvyC4SE+Wt3M5iCGXwQVoFRTrJpCEpfBW3iBHjBQFKbFC9uK1lPqpX95dj1Ll0HyD3jOUtbx8LlBPwAEwuevS8ARTbyTqMkiCYew503iq3vECOpwcnSU2WvEayEs7Ah5ulsHfgyi3A2T2GoAfXN1SW9pIbCNAxI9VLJIp8nI+PUflVDsmAgYIty+YkIfz7YMXbVmcrd4ZlFF7bBGbojpa+WOM+z2/eAgcEolmWz91BD7xbdHz6DxB9pA0hMPEAMCGTQ412b4inFS/avThn/D+HW2QHUFhE4A8vMBBkEG5l13UyI0cqFGtorOJSMjAGYcODAAHBotiBdTBFWC2TozM+ZeCqAmExYUhaoyd/r2cV8tM72l1s8AuGsrY7wRgD6efpetBpHsy9rJdx9Rt38ISe4gdGZtMnBJyRW/lVoTWY9DNd8yTSMzbtqLGeilkEJTPzZVGlykvU2PMirr8f1fY1ovXO4YGeubt58KSc7b18SLI9qtgFT4FY6QR2zdKZ/qY8hy8xL3QcbPTmQFDz2P0HFQ3Oyj7blTF6MPQQofkDmyHHC7+k31uiUYCmHVlX0UcpK8aeOIVYYrWZyDXVdAgRxZmbzFrdWHOWYZvIbDAGopCRtoonewC64xfTEnRBr6T9Lk7bBuQtHHXIi8ZcoaWooxA/7ViNKAhFo7e92lNIxzJQs1j/4SA X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: ac137967-caca-4d11-74a5-08dd2e72ec98 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jan 2025 16:55:28.8328 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ICOzl3PPXsR8+AsCjFXoRe0oS188z61sUlHigfmTcPhWncvLmDbVk6xPfFs+PpLh X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6634 X-Rspamd-Queue-Id: 549D8160021 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: mie5c3xie1cq967cy7nh744z1nxpa6og X-HE-Tag: 1736182554-124194 X-HE-Meta: U2FsdGVkX1/DORtjM2Ot5kz2jXa4tbLT4hdzxsAZu3JWuHMhNX0rDc2yAbv/C+80WDwHOmie7RnJsdFhVn3UrHSa5m8qL9rz/CBohHH6l1FCXfToaNsk/ZFTWO0xoIoKiZFEqyifLRJdh3Vhu7GnQQ+FmJ0mwE9ZTRhlj0N21eh4hrqb4DWMqxDqwriI3HWckPFDXAjzFrzXLOzRMGlugRgesKuDJipLe6xBN6ryfIof/xMSb7As/JJCoPNjfrf4rDz7l2IZgSmQ7ruxqEnSXIFDOakj8p2v4PP1hKcpXNooyK125JTkkhUd9Fupr9UmpfOf6zXWUTdPTPxdg8E/6O9PwfBlUs+ltSfBYkS8EauomhVv7UTkagY5mdTSY271M/oZWkCsgIY9aJlMIONEV/ckE5+Ckzq32/gCDa436VUtpDnc5/8JJ4hOjVeEFUrX2ARfleKHWj1IXR4Ve+zkOYStbsCT/TvyjYPHOdFl5O+nlOCzXGq0Ofh/kezXXBFYYBBkANrqMS5AohVIpIu9M7P1BV0gTouYv5uZmSOyp1/puoBid7g0IR1/jJXEwpfhXMxNpUPbHqrje9MkROPOQpc1a8Fv9UTLzzEx4RfncN8U7O9cxD+G25rS1lIMcdcX9mFMOV/3OW0NkxaixQuJz1Jn0+FC9CrEpbaceDBY+Q7bt/57nAG6cM1lMJsZuq0qlaEpyJOYNGjpJh8gDIQ8iL8D0pyNPfmvLzPKSBpTMpBeardWGsz7tCArgAyJnd4mrPeBIfPbpOEqJ5mU7UWD8ODduLMYuTWudOD+OfGwaAO+LGijkCgVcewSf3ERi4hgwj4ykntHnEAGBlR5xAp/Nj6wedmbDAfbiES4Wfgl6Ceb1YoS1MOhtW1LKuL8LsupmDutBTKP8ijNLQYvJ90DC3ywLLW8XonVvMvG5Av+yBmg3gRR3WCDnOBbAuMJo2wVAUQk9UfTMVduvVj5vbu bTbVcFKj gLaQaJ1glsRfOuLUpt4Ts1yf1rVuY6wls489w102ZeJxArKn9ACQj+PNUm74xbOvsPNFL2jfOO5kvWqx4OQV+fPVPPxVX7Rf9U4C6piTtiqjFgc6XqYJfGsAs7EIGf1TKCIjzPi1HaGTCnw4zQNZn1TSXfKw7KsgW/hBkUiJczijclY4wjFi5s3jxgBhn7HGDX0iNcuIfCgCNkCF5BuJUZF98yXCTYWzdtIcb7gDUJNuoGown2R7YNy7f8ku/J54rrc9T8RAkTbZlvLIl3KQu4nKrxn2MhsmnKPPI62UIYwHzU4NdKzZBLzoOl+Niz1tRiW6jzyZi0bJLngsKnK20GxpOV7I3Am1aRyxORzHdgcGdzpbFBYbnn7Rynk3SSuyMrARv9QQrRfhl62o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: folio_split() splits a large folio in the same way as buddy allocator splits a large free page for allocation. The purpose is to minimize the number of folios after the split. For example, if user wants to free the 3rd subpage in a order-9 folio, folio_split() will split the order-9 folio as: O-0, O-0, O-0, O-0, O-2, O-3, O-4, O-5, O-6, O-7, O-8 if it is anon O-1, O-0, O-0, O-2, O-3, O-4, O-5, O-6, O-7, O-9 if it is pagecache Since anon folio does not support order-1 yet. It generates fewer folios than existing page split approach, which splits the order-9 to 512 order-0 folios. folio_split() and existing split_huge_page_to_list_to_order() share the folio unmapping and remapping code in __folio_split() and the common backend split code in __split_unmapped_folio() using uniform_split variable to distinguish their operations. Signed-off-by: Zi Yan --- mm/huge_memory.c | 58 +++++++++++++++++++++++++++++++++--------------- 1 file changed, 40 insertions(+), 18 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 586870e60003..e5f70ff88a1a 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3725,11 +3725,10 @@ static int __split_unmapped_folio(struct folio *folio, int new_order, } static int __folio_split(struct folio *folio, unsigned int new_order, - struct page *page, struct list_head *list) + struct page *page, struct list_head *list, bool uniform_split) { struct deferred_split *ds_queue = get_deferred_split_queue(folio); - /* reset xarray order to new order after split */ - XA_STATE_ORDER(xas, &folio->mapping->i_pages, folio->index, new_order); + XA_STATE(xas, &folio->mapping->i_pages, folio->index); bool is_anon = folio_test_anon(folio); struct address_space *mapping = NULL; struct anon_vma *anon_vma = NULL; @@ -3750,14 +3749,15 @@ static int __folio_split(struct folio *folio, unsigned int new_order, VM_WARN_ONCE(1, "Cannot split to order-1 folio"); return -EINVAL; } - } else if (new_order) { + } else { /* * No split if the file system does not support large folio. * Note that we might still have THPs in such mappings due to * CONFIG_READ_ONLY_THP_FOR_FS. But in that case, the mapping * does not actually support large folios properly. */ - if (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && + if ((!uniform_split || new_order) && + IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && !mapping_large_folio_support(folio->mapping)) { VM_WARN_ONCE(1, "Cannot split file folio to non-0 order"); @@ -3766,7 +3766,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order, } /* Only swapping a whole PMD-mapped folio is supported */ - if (folio_test_swapcache(folio) && new_order) + if (folio_test_swapcache(folio) && (!uniform_split || new_order)) return -EINVAL; is_hzp = is_huge_zero_folio(folio); @@ -3823,10 +3823,13 @@ static int __folio_split(struct folio *folio, unsigned int new_order, goto out; } - xas_split_alloc(&xas, folio, folio_order(folio), gfp); - if (xas_error(&xas)) { - ret = xas_error(&xas); - goto out; + if (uniform_split) { + xas_set_order(&xas, folio->index, new_order); + xas_split_alloc(&xas, folio, folio_order(folio), gfp); + if (xas_error(&xas)) { + ret = xas_error(&xas); + goto out; + } } anon_vma = NULL; @@ -3891,7 +3894,6 @@ static int __folio_split(struct folio *folio, unsigned int new_order, if (mapping) { int nr = folio_nr_pages(folio); - xas_split(&xas, folio, folio_order(folio)); if (folio_test_pmd_mappable(folio) && new_order < HPAGE_PMD_ORDER) { if (folio_test_swapbacked(folio)) { @@ -3905,12 +3907,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order, } } - if (is_anon) { - mod_mthp_stat(order, MTHP_STAT_NR_ANON, -1); - mod_mthp_stat(new_order, MTHP_STAT_NR_ANON, 1 << (order - new_order)); - } - __split_huge_page(page, list, end, new_order); - ret = 0; + ret = __split_unmapped_folio(page_folio(page), new_order, + page, list, end, &xas, mapping, uniform_split); } else { spin_unlock(&ds_queue->split_queue_lock); fail: @@ -3988,7 +3986,31 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, { struct folio *folio = page_folio(page); - return __folio_split(folio, new_order, page, list); + return __folio_split(folio, new_order, page, list, true); +} + +/* + * folio_split: split a folio at offset_in_new_order to a new_order folio + * @folio: folio to split + * @new_order: the order of the new folio + * @page: a page within the new folio + * + * return: 0: successful, <0 failed (if -ENOMEM is returned, @folio might be + * split but not to @new_order, the caller needs to check) + * + * Split a folio at offset_in_new_order to a new_order folio, leave the + * remaining subpages of the original folio as large as possible. For example, + * split an order-9 folio at its third order-3 subpages to an order-3 folio. + * There are 2^6=64 order-3 subpages in an order-9 folio and the result will be + * a set of folios with different order and the new folio is in bracket: + * [order-4, {order-3}, order-3, order-5, order-6, order-7, order-8]. + * + * After split, folio is left locked for caller. + */ +int folio_split(struct folio *folio, unsigned int new_order, + struct page *page, struct list_head *list) +{ + return __folio_split(folio, new_order, page, list, false); } int min_order_for_split(struct folio *folio)