From patchwork Fri Nov 1 15:03:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13859513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2995E6F063 for ; Fri, 1 Nov 2024 15:04:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 198866B0088; Fri, 1 Nov 2024 11:04:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0D7306B0095; Fri, 1 Nov 2024 11:04:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DAB1C6B0098; Fri, 1 Nov 2024 11:04:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A119C6B0088 for ; Fri, 1 Nov 2024 11:04:21 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 571BB160731 for ; Fri, 1 Nov 2024 15:04:21 +0000 (UTC) X-FDA: 82737846240.10.C14C691 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2061.outbound.protection.outlook.com [40.107.220.61]) by imf11.hostedemail.com (Postfix) with ESMTP id 9534940035 for ; Fri, 1 Nov 2024 15:03:43 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=LSccG+AY; spf=pass (imf11.hostedemail.com: domain of ziy@nvidia.com designates 40.107.220.61 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730473279; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eJ7mtvl5+aRqSZmDpJAAE0enIUlEdEtuMZ5VoenL7co=; b=G1aW9QrCIVt+hQ8ZyDWVw2z/UpERiWxljZLDFwFGto37C0YNO66Ji+y0d4JaeqjlgWw7T2 20gjjntW+0IazqISggqSTPuiqfcFPa6pozqzv9jzRgL9IKFxgKf7QZSn3KUXUTy9cx5jZr CBJWOAQApv+FsCQzE/unqLdNFJkNxwQ= ARC-Authentication-Results: i=2; imf11.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=LSccG+AY; spf=pass (imf11.hostedemail.com: domain of ziy@nvidia.com designates 40.107.220.61 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1730473279; a=rsa-sha256; cv=pass; b=6boEXBadF/00sYIHi06OCDoepRUnY9uY+dRp7mJakyonov9I9BaD8tzW4TtGnRufRWy+zK gjmk0q+zv+fdq9qFRZvYffWFJ9311GzFCmXkblepmsAeSFfJyMCPvU5Co8/QF+6vZ2UcPx M9oD93SbKjcU3xSDuWaBpgDKkPtYkZw= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=oHVz29DmmK6auSY3sMz9Qh+l8gmdgEDZKlgfVhikODDLaoNPWE3k7WHJekV4hSYTZuE864ZhEDc6mqlaCwFPRh/x9oZvDKLPKmQFzCc6ufydaQW4XTT5mlQ8Kz8k/sg/jlHKHoJIhCM8sxbFsVmGWj8LPmU01zLg75GmMdybJs+lBnuAzPRjvwmV6GMizC0QhxxOz6c9Gk3my97guqN0ki7FF4ugojSqwHAGOS6KxAhByZU7QVFy7O3KjPQtCLRM0y2jaTKD7YITPZtXBX7IKxSN+hC3pwMHAU0bNsXUtJRv+DOgMzbiIFY9OkQDTNjbfZ+9fxdmI/yndZTinyTtVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eJ7mtvl5+aRqSZmDpJAAE0enIUlEdEtuMZ5VoenL7co=; b=fWzUWqZRQnqTQHDe85hGeqtyp1TRjT4G69iPXVRQOkn3ahq7AopW7j5lHoQlF63/TJs/YdzK2SyIuRYdj7Y4k/QzaowQBEqbe/davsjIirZcwQIVA34xqE2zIw+Q+9YCA4yBHOT+Pq5q94kTGFmSU7N6I+JwJdUgaH0JaZ2ezHvM7I+WwrpxO4c8uGMabztfVaHUfyNyMfwoREuzjuONKKOj6q60zBGoZR7Z2mtIlGv8/QZzPPDvJMYGfgK0FlrNK4XkP9x4epIgQUiJPCAtqX+ylJ58NwZSMDufIxnexBSN8glWs+LxR39SeFu1Uu0MYRyqo/X37dT7On7eaSiI8g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eJ7mtvl5+aRqSZmDpJAAE0enIUlEdEtuMZ5VoenL7co=; b=LSccG+AYEttToFeO0f+mF13iWqqGGEQj0x2F8D1elSMexwrfO9RuzKzPCNKKVlRgoPZGcdj6yd9GqeryQQCNIdddKdQT3arH7Kab0NgEXUE8EF6IO3BcdmHKOeULrPBStpbp/f/xeCinytKYUXZABj5WsQmZ5Yk5Q+4xstI+MJK91NQgNmtpeWkRPCE5UmbZUfB7qpGruLTn2JzgIX9QTck7byYQ/kkUlLxFZVvzqGO/bYgFFJYJCI7tN4hMTkgcFkpEYbIEGrgDXULeQhya7XEU8O6GINNwTOj2bkqaS0S85ogzzBaHL5hbW8qOucgwNX9DnJ7n73TRGv0GJMqIWQ== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CH3PR12MB9453.namprd12.prod.outlook.com (2603:10b6:610:1c9::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8114.18; Fri, 1 Nov 2024 15:04:08 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%7]) with mapi id 15.20.8114.015; Fri, 1 Nov 2024 15:04:08 +0000 From: Zi Yan To: linux-mm@kvack.org, "Kirill A . Shutemov" , "Matthew Wilcox (Oracle)" Cc: Ryan Roberts , Hugh Dickins , David Hildenbrand , Yang Shi , Miaohe Lin , Kefeng Wang , Yu Zhao , John Hubbard , linux-kernel@vger.kernel.org, Zi Yan Subject: [PATCH v2 1/6] mm/huge_memory: add two new (yet used) functions for folio_split() Date: Fri, 1 Nov 2024 11:03:52 -0400 Message-ID: <20241101150357.1752726-2-ziy@nvidia.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241101150357.1752726-1-ziy@nvidia.com> References: <20241101150357.1752726-1-ziy@nvidia.com> X-ClientProxiedBy: BN9PR03CA0109.namprd03.prod.outlook.com (2603:10b6:408:fd::24) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CH3PR12MB9453:EE_ X-MS-Office365-Filtering-Correlation-Id: 681b0ca5-bfe9-42c1-9813-08dcfa866f9f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|376014|1800799024; X-Microsoft-Antispam-Message-Info: SkIMXVg5P3jbKaLprVOdyJ+Y2Z0AMp+iAnXsp83b2L5381qlPrcm3TXbxBfRfR66ZgC+Jjpt6YQ5qZq7r4Sq2Lyvw/hWW5axeC8KmTgIfVySd7WjeMSY215eLPi7fJR1V2StsYgnlLKNRP9wLmCOogcQ7hYI90p+pKk7FQxzFmUK/0/h0TYn3rsiG9rHjUlCGwaOs7RZQip7POlP8C4AFIm1/0qS9UalLadnCsR6KyJKuUqmpYlYQgugOmkFFUVgvcmYHNWEB769cosimwdPgelNlWAKFhLH2YwD6vZF/3/rPOTzzlHBnsKYTFI4zTQQFGhgIo6ehQ+dvkykbw5k+GUWEdVL6Cbtirx3bDbc1oHVUuBqD+FFQHUZ+xQhJ0eLAc3mrSEZgmuXhwy6/abmkQarBbgPpu28EJgWbb7l4XtT+pYFFoF9zEiY6EjXTgAeWT4WyrBa/ZwuW3az8IgD+7Noe4E6dbWRBtdi7MoivSD+7XYIW0ARSCBzXd6Bqv8zk0mB5JGgF4wIvMQyuopEDXn9uOlJDbBeAOmxVQiXpT7GrPlv9XUG+FLZ8KineiFabwqThpRMB5S1SAD5VqmpH04yHZtu3akR3xp36tKSi7vfFjTyFMrhZs381MjZY0/V6whsauiBhbeOWMWrQM8C7mjLtoEIuBhmtBSDWwpREiDfh+h8Hu0T3L3CjQyl289+PwnGVg9URUJRIivR3qqChyAuXhQMZArNIojl6BRbDlJYAB0iq2kExa86kNmlF93vwvNzIrrOOWYOp2tJPe9iudC8uZftvVdgj3YON8dnHEimaValUyGN1BLpC52uB11pJQDcAPsQu2Uy5Bz31/IZWCxkNTWpRLYb37fjoOElM1lw3Yix2QchoYLFVaPjWll92mhWt5djYZnirUPHaIXKwVJNC4TTkm+0Tf5k3pfE8ZolwQXf1zBeK6mp9O0/zZoIiSx9aL+zFS/uydxgSraJkVem2DVY7mswB1gFAAlcSdyLlUYw8CXsF9oS0c4hF2zuY/OjuwTwa4saTH+6Mkmc8em+KJkKSrec8JCmj2SQ9Am7M8t1WgArfvzeyWc0RgufdKPFWfj3pn3B5zGPMlggLkwVU+dJuZD/HZjNeU6okROdpEJu/N9vFMNrwJPqnXcFM03imgjuJ1WEoA5u+K7NzyL0djVoaq++ZktAtp88GNKl+t77EPRET/uSHVF1vGUxwULpcuKdLmhWyv6ULqkIgta0hkNR9iyp9tkqd0LdTgqDnA+BZPMFoiegnTzPduCTzFhz3boHXpNRjBvteWOGX1LracXKGCyn++oe1gxyezsKyEJHBXnCw2wACiY/7WsM X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(7416014)(376014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Qnp1EXSvt4oAOH81UHBiKlnSm8eZJSN29g6q73v6oUv8jtHHKVfmJGYR/ZnaOWh43nLtoO1zdExsd7JfAMYeS3ZTPonJoDXIdcX1/WtaE7YbCw+w8CAUFVvRf6S5viXxuKveOrbWaDLiZP+XBp/QvQ+X2YSMs79NxOBT3b+S0vgBxda2RFAZM83kthAzMTFx4F79IBQ4OpQZnr9zctfOJ3guDDMq1YlvGLjAxOrKtQWh+CePBvhSqMmWH3cdispeP6DFcPsbgcFIV1e7j6fYLfhfv005JpLt5f3d7BMe6EGUmrhC/RceFQq/Dc/tYVjNdlCTXVUlzQBvMikv0ApP9QX7LRlRqvtnaJT0HWBfRk6XPKr/5ATAgcijQO7q3O+2WfFC8oYcS+tcBUXkIBq9r8/ynXGsl/qDLIyZfKXSpPTAoyuWP5vkpCRldht0vpqDbw+cf+SPwYFuJU5UbHiqlEZKmkqdnguSdMsNrh3l3KpMHvU7i3nMyaEm6eBGoOsI4S6W6bCPqtMBHmm9JraAX0IIyUQrABIwi2ip0l5FY3TfCkAnqx2Exldqwsq8rRV9b6ewOwRr7adsKuQ8ERmgrHqwacRNOc7TrNM4oK09171qKxXgkzSXipMA/Cii4SHcL8hAExoKdJQ9+nNd3uGfgWg49hL07FXEnkSD/GIjkfgwIZI/NaPFnvb5SIX9KcTyx1lAFlyH5TV6yJ6vxBwOW6b+oRiUnN/qFEzX3y2ZZ5cNEB/d2rmX7qqz6tnXvS4A1MZtlTUO3FaC8tKem99Ps8AKl27iBm1iJU4IIEdeJIbv+1/TQSzK8MOGouMb2I1feikWol1KSpdDkLTMFPPDVHJwOOHZ5BJRJzOyhm4vh23ifaMWUD0w/logbGap6NSyhVrb6fDcXp+sCTFKx72uNmXuw2F9laBHLF85G7yeSPcqrONPir4HgSWuurdyq9lCGn3/jDwZg6Mnl5f1hdSHQhp+DfwMtsjm10ClgOZzxczxYa7/dXpG7ccDEaPOHG5ZNBSeOuhdoiNuL9/qdh0Wwwa0RbLO+ILUFOVVxWxVWwN7rsz72An+D+PInJXAaroMS8C6bC4wWpOBymAwoSaJ0/hFmtONeSh1SpA+We/Fm9Qivl31wUvSZGlBoZy9avZSHA0kjAs0j6xCVXRGZYotxUq5BFEN7dN7oFWPzLdT7bg0yMDp81oSkGhSsmcZEPSFxV0Ehjt4F3FYAzrkH0Opw1SDtXVfpa4+RKAbG4QQDCjFfPJXlefZDF0xIsMitVhkLa1ZYTXtDggVDAUSU368HHFajFcj0C4rC1hj7B58QE7LHaplwW9Pb9h5Lto1mIVuJolL0Dnz5/clfH5f99OTdl3HHLgN5ohVWHYJpJStEuO1XmzA8h4FRCgBekjLU31WEQiqu5R/Q/geDaGJodD8oUIhnADWV2/FwNBAcV9OBRxYo+xbp1CPUvxdfAqqNpi5UMmD78x70iVTyyBHJQzOneHgdY8cF3f63j5zoiDIwUqC6LhZmPS7supr8LPynq3Ayh++cOyAgpRUBrTMg7GUoWsNO79Om6usN/m2LzHrrKv9Klv8kC/1aLaah2BewqZR X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 681b0ca5-bfe9-42c1-9813-08dcfa866f9f X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Nov 2024 15:04:08.6370 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: PD/jTcoBynkQMgMF/tkvN8EHQdzLdfxV5C/XloNNzL+OaNZJKniqep0GnGEleZcu X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB9453 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 9534940035 X-Stat-Signature: swj4pearfbxwg19pwe86wqbf4fkde6cf X-Rspam-User: X-HE-Tag: 1730473423-30546 X-HE-Meta: U2FsdGVkX1+O1oQ5KKc1VO3Heo9eiBGSC89qxifotf9glwawZy7vVJCc7LEUMSZRE54N/XfGYyEUJ7Vqd1gvIP/4b25ZpZUtGseF/VLbaWZEgwEyEHS8MMPqaZF8Sn6QEmvn8gzdDg3MQvjwrl3ThfMIG+iM+0+u1QyPg/YqqpJBZq40vo1JE5DxFjzOvGT81yf5n23Dv8UPZBRNd2JUMCmeOYFQEyooUPO8DfdoDln0B9BeEi/e/kbgHBXakOjtlJlbKwSOji9R337vWqujdPPUbhbq1hYGg6BFPOSPyC3MG6XE7aXD5VQxrmQiJTKrdi3zlnT1CKXFvj4i6UAOVHjv73BgXIKCM1GwQ6/T0ERsrr3P7hlV+j6K6Nvk3bLs9p3QHXl3wnMYTSjopwAKkeIIZv2nNtsAxZFn8iQgHnerMTOCCmAQjb2xxIHgTFYYjwLQepVb+7mCeOKcMM1aytow6iLV8UUB4fFRfqBn4xnhdCGi02jMQH0mg7XzS1UPmpbA4dSEQaCHb5NysAboJwzcGuB0oSHONDFrHhzmch5BUZx5T+7z7pAoMjYhNOjQfxQc8gKBKA3WHU9hQDr4oBfk+nZnttaDe5gRTK/H6hd78qhJbSzRR3IJ8Y9+WWKWvAFn8Uhzhyk15V1/90WeqiQNghBT1UYanRxLbZ6vbYq3zFm4HcTNKkN+C8KOowKxU75WuSggCwEo3X66nepAClXBqM+OcpEPpGoaroemmwrtvor9TAegBGB1tHoO+K3GjxjGpQOnyaAa5eJAiORXS5eqqhDP9Y3MIG1kskDuHVMkeQc5sDH3w+m0eqL7moovyHRjfus9gizKu5xqKBimojeoOweklKgKYCSZgKANk4woli9eRjlNVwbcjwk2SUlH1GdzgXOdUF28AdF9NAE21Eb1i263nv/zjmEnT/ldMUnOssWF9ikx6mNcFocGBBOoMAxYx89kcIo9tYKG5WP afbHD1B3 uXrQyhMuitEZ7zntk4jsnZEVQx8u8sChh2XGlKwD+5Nny8oYSDzwsBYV7MaT6cDL265qqw2EvuF6noPN58RVx1iAeJb2wdUaGgnrNMqTQRlJwZGQivJWvot+r9ZvQbIfb9czcEcObUubmZQK3e7pNXMi9qBUxR7ZUDtXfqmpRzDwfYw54HM2Th9w4omdJD/xc7f3wVKCptykgJDnuviOcPOiAeEZjO/EyvOyKVv8fm0ZpfKHY3OOVQQWWeuzOrLgpYXlAcie4MNZ5/cTEM5hDFk+idLGf1SD7fi/s/n6UmB3b7c9/3o7rCv0b7zGupPAQEMnLnb0e4fYIztRZG9Zcgux0cw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation patch, both added functions are not used yet. The added __folio_split_without_mapping() is able to split a folio with its mapping removed in two manners: 1) uniform split (the existing way), and 2) buddy allocator like split. The added __split_folio_to_order() can split a folio into any lower order. For uniform split, __folio_split_without_mapping() calls it once to split the given folio to the new order. For buddy allocator split, __folio_split_without_mapping() calls it (folio_order - new_order) times and each time splits the folio containing the given page to one lower order. Signed-off-by: Zi Yan --- mm/huge_memory.c | 328 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 327 insertions(+), 1 deletion(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index f92068864469..f7649043ddb7 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3135,7 +3135,6 @@ static void remap_page(struct folio *folio, unsigned long nr, int flags) static void lru_add_page_tail(struct folio *folio, struct page *tail, struct lruvec *lruvec, struct list_head *list) { - VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); VM_BUG_ON_FOLIO(PageLRU(tail), folio); lockdep_assert_held(&lruvec->lru_lock); @@ -3379,6 +3378,333 @@ bool can_split_folio(struct folio *folio, int caller_pins, int *pextra_pins) caller_pins; } +static long page_in_folio_offset(struct page *page, struct folio *folio) +{ + long nr_pages = folio_nr_pages(folio); + unsigned long pages_pfn = page_to_pfn(page); + unsigned long folios_pfn = folio_pfn(folio); + + if (pages_pfn >= folios_pfn && pages_pfn < (folios_pfn + nr_pages)) + return pages_pfn - folios_pfn; + + return -EINVAL; +} + +/* + * It splits @folio into @new_order folios and copies the @folio metadata to + * all the resulting folios. + */ +static int __split_folio_to_order(struct folio *folio, int new_order) +{ + int curr_order = folio_order(folio); + long nr_pages = folio_nr_pages(folio); + long new_nr_pages = 1 << new_order; + long index; + + if (curr_order <= new_order) + return -EINVAL; + + for (index = new_nr_pages; index < nr_pages; index += new_nr_pages) { + struct page *head = &folio->page; + struct page *second_head = head + index; + + /* + * Careful: new_folio is not a "real" folio before we cleared PageTail. + * Don't pass it around before clear_compound_head(). + */ + struct folio *new_folio = (struct folio *)second_head; + + VM_BUG_ON_PAGE(atomic_read(&second_head->_mapcount) != -1, second_head); + + /* + * Clone page flags before unfreezing refcount. + * + * After successful get_page_unless_zero() might follow flags change, + * for example lock_page() which set PG_waiters. + * + * Note that for mapped sub-pages of an anonymous THP, + * PG_anon_exclusive has been cleared in unmap_folio() and is stored in + * the migration entry instead from where remap_page() will restore it. + * We can still have PG_anon_exclusive set on effectively unmapped and + * unreferenced sub-pages of an anonymous THP: we can simply drop + * PG_anon_exclusive (-> PG_mappedtodisk) for these here. + */ + second_head->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; + second_head->flags |= (head->flags & + ((1L << PG_referenced) | + (1L << PG_swapbacked) | + (1L << PG_swapcache) | + (1L << PG_mlocked) | + (1L << PG_uptodate) | + (1L << PG_active) | + (1L << PG_workingset) | + (1L << PG_locked) | + (1L << PG_unevictable) | +#ifdef CONFIG_ARCH_USES_PG_ARCH_2 + (1L << PG_arch_2) | +#endif +#ifdef CONFIG_ARCH_USES_PG_ARCH_3 + (1L << PG_arch_3) | +#endif + (1L << PG_dirty) | + LRU_GEN_MASK | LRU_REFS_MASK)); + + /* ->mapping in first and second tail page is replaced by other uses */ + VM_BUG_ON_PAGE(new_nr_pages > 2 && second_head->mapping != TAIL_MAPPING, + second_head); + second_head->mapping = head->mapping; + second_head->index = head->index + index; + + /* + * page->private should not be set in tail pages. Fix up and warn once + * if private is unexpectedly set. + */ + if (unlikely(second_head->private)) { + VM_WARN_ON_ONCE_PAGE(true, second_head); + second_head->private = 0; + } + if (folio_test_swapcache(folio)) + new_folio->swap.val = folio->swap.val + index; + + /* Page flags must be visible before we make the page non-compound. */ + smp_wmb(); + + /* + * Clear PageTail before unfreezing page refcount. + * + * After successful get_page_unless_zero() might follow put_page() + * which needs correct compound_head(). + */ + clear_compound_head(second_head); + if (new_order) { + prep_compound_page(second_head, new_order); + folio_set_large_rmappable(new_folio); + + folio_set_order(folio, new_order); + } else { + if (PageHead(head)) + ClearPageCompound(head); + } + + if (folio_test_young(folio)) + folio_set_young(new_folio); + if (folio_test_idle(folio)) + folio_set_idle(new_folio); + + folio_xchg_last_cpupid(new_folio, folio_last_cpupid(folio)); + } + + return 0; +} + +#define for_each_folio_until_end_safe(iter, iter2, start, end) \ + for (iter = start, iter2 = folio_next(start); \ + iter != end; \ + iter = iter2, iter2 = folio_next(iter2)) + +/* + * It splits a @folio (without mapping) to lower order smaller folios in two + * ways. + * 1. uniform split: the given @folio into multiple @new_order small folios, + * where all small folios have the same order. This is done when + * uniform_split is true. + * 2. buddy allocator like split: the given @folio is split into half and one + * of the half (containing the given page) is split into half until the + * given @page's order becomes @new_order. This is done when uniform_split is + * false. + * + * The high level flow for these two methods are: + * 1. uniform split: a single __split_folio_to_order() is called to split the + * @folio into @new_order, then we traverse all the resulting folios one by + * one in PFN ascending order and perform stats, unfreeze, adding to list, + * and file mapping index operations. + * 2. buddy allocator like split: in general, folio_order - @new_order calls to + * __split_folio_to_order() are called in the for loop to split the @folio + * to one lower order at a time. The resulting small folios are processed + * like what is done during the traversal in 1, except the one containing + * @page, which is split in next for loop. + * + * After splitting, the caller's folio reference will be transferred to the + * folio containing @page. The other folios may be freed if they are not mapped. + * + * In terms of locking, after splitting, + * 1. uniform split leaves @page (or the folio contains it) locked; + * 2. buddy allocator like split leaves @folio locked. + * + * If @list is null, tail pages will be added to LRU list, otherwise, to @list. + * + * For !uniform_split, when -ENOMEM is returned, the original folio might be + * split. The caller needs to check the input folio. + */ +static int __folio_split_without_mapping(struct folio *folio, int new_order, + struct page *page, struct list_head *list, pgoff_t end, + struct xa_state *xas, struct address_space *mapping, + bool uniform_split) +{ + struct lruvec *lruvec; + struct address_space *swap_cache = NULL; + struct folio *origin_folio = folio; + struct folio *next_folio = folio_next(folio); + struct folio *new_folio; + struct folio *next; + int order = folio_order(folio); + int split_order = order - 1; + int nr_dropped = 0; + int ret = 0; + + if (folio_test_anon(folio) && folio_test_swapcache(folio)) { + if (!uniform_split) + return -EINVAL; + + swap_cache = swap_address_space(folio->swap); + xa_lock(&swap_cache->i_pages); + } + + if (folio_test_anon(folio)) + mod_mthp_stat(order, MTHP_STAT_NR_ANON, -1); + + /* lock lru list/PageCompound, ref frozen by page_ref_freeze */ + lruvec = folio_lruvec_lock(folio); + + /* + * split to new_order one order at a time. For uniform split, + * intermediate orders are skipped + */ + for (split_order = order - 1; split_order >= new_order; split_order--) { + int old_order = folio_order(folio); + struct folio *release; + struct folio *end_folio = folio_next(folio); + int status; + bool stop_split = false; + + if (folio_test_anon(folio) && split_order == 1) + continue; + if (uniform_split && split_order != new_order) + continue; + + if (mapping) { + /* + * uniform split has xas_split_alloc() called before + * irq is disabled, since xas_nomem() might not be + * able to allocate enough memory. + */ + if (uniform_split) + xas_split(xas, folio, old_order); + else { + xas_set_order(xas, folio->index, split_order); + xas_set_err(xas, -ENOMEM); + if (xas_nomem(xas, 0)) + xas_split(xas, folio, old_order); + else { + stop_split = true; + ret = -ENOMEM; + goto after_split; + } + } + } + + split_page_memcg(&folio->page, old_order, split_order); + split_page_owner(&folio->page, old_order, split_order); + pgalloc_tag_split(folio, old_order, split_order); + + status = __split_folio_to_order(folio, split_order); + + if (status < 0) + return status; + +after_split: + /* + * Iterate through after-split folios and perform related + * operations. But in buddy allocator like split, the folio + * containing the specified page is skipped until its order + * is new_order, since the folio will be worked on in next + * iteration. + */ + for_each_folio_until_end_safe(release, next, folio, end_folio) { + if (page_in_folio_offset(page, release) >= 0) { + folio = release; + if (split_order != new_order && !stop_split) + continue; + } + if (folio_test_anon(release)) + mod_mthp_stat(folio_order(release), + MTHP_STAT_NR_ANON, 1); + + /* + * Unfreeze refcount first. Additional reference from + * page cache. + */ + folio_ref_unfreeze(release, + 1 + ((!folio_test_anon(origin_folio) || + folio_test_swapcache(origin_folio)) ? + folio_nr_pages(release) : 0)); + + if (release != origin_folio) + lru_add_page_tail(origin_folio, &release->page, + lruvec, list); + + /* Some pages can be beyond EOF: drop them from page cache */ + if (release->index >= end) { + if (shmem_mapping(origin_folio->mapping)) + nr_dropped++; + else if (folio_test_clear_dirty(release)) + folio_account_cleaned(release, + inode_to_wb(origin_folio->mapping->host)); + __filemap_remove_folio(release, NULL); + folio_put(release); + } else if (!folio_test_anon(release)) { + __xa_store(&origin_folio->mapping->i_pages, + release->index, &release->page, 0); + } else if (swap_cache) { + __xa_store(&swap_cache->i_pages, + swap_cache_index(release->swap), + &release->page, 0); + } + } + xas_destroy(xas); + } + + unlock_page_lruvec(lruvec); + + if (folio_test_anon(origin_folio)) { + if (folio_test_swapcache(origin_folio)) + xa_unlock(&swap_cache->i_pages); + } else + xa_unlock(&mapping->i_pages); + + /* Caller disabled irqs, so they are still disabled here */ + local_irq_enable(); + + if (nr_dropped) + shmem_uncharge(mapping->host, nr_dropped); + + remap_page(origin_folio, 1 << order, + folio_test_anon(origin_folio) ? + RMP_USE_SHARED_ZEROPAGE : 0); + + /* + * At this point, folio should contain the specified page, so that it + * will be left to the caller to unlock it. + */ + for_each_folio_until_end_safe(new_folio, next, origin_folio, next_folio) { + if (uniform_split && new_folio == folio) + continue; + if (!uniform_split && new_folio == origin_folio) + continue; + + folio_unlock(new_folio); + /* + * Subpages may be freed if there wasn't any mapping + * like if add_to_swap() is running on a lru page that + * had its mapping zapped. And freeing these pages + * requires taking the lru_lock so we do the put_page + * of the tail pages after the split is complete. + */ + free_page_and_swap_cache(&new_folio->page); + } + return ret; +} + /* * This function splits a large folio into smaller folios of order @new_order. * @page can point to any page of the large folio to split. The split operation