From patchwork Tue Nov 5 03:29:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Hubbard X-Patchwork-Id: 13862315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7633D1CA15 for ; Tue, 5 Nov 2024 03:30:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A7276B00A2; Mon, 4 Nov 2024 22:30:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 357336B00A3; Mon, 4 Nov 2024 22:30:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D07F6B00A4; Mon, 4 Nov 2024 22:30:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id EC6556B00A2 for ; Mon, 4 Nov 2024 22:30:08 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 6384E1C419A for ; Tue, 5 Nov 2024 03:30:08 +0000 (UTC) X-FDA: 82750611972.12.AF5D42F Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2062.outbound.protection.outlook.com [40.107.100.62]) by imf06.hostedemail.com (Postfix) with ESMTP id 55018180021 for ; Tue, 5 Nov 2024 03:29:42 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=hNSwAwKW; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf06.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.100.62 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730777184; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6SMMUmKUWMrpwMdn6et7YI8Q82K0AaYpk74A+1JAtSw=; b=3gYr7+mcavA860ZjRKhf0SzkcLCCw9+ZnprdCWxtzG11sSjnEbpK4foEli4nIumPdVEdaF 25XA6hcoohVI/gP3E8tsEKPnEItoNAi7NE/Ir4Tlvd4N9kRYIkczhyhgwc48nbA4NeRTYE +D5uIWpYqgiFXsAH6GiJ7iDp4BhtL8Q= ARC-Authentication-Results: i=2; imf06.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=hNSwAwKW; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf06.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.100.62 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1730777184; a=rsa-sha256; cv=pass; b=P9IY/r5vhNyDHN/jXmCCSybHMGuOUONjvWkLrXj1ylKLMGCN1SxOZyuT182lVY3ZV1H0F/ 2KKOYrp7rDRxYmv74qIZTt+KPq1G6O4WkF7vA6Oqvc4mxDA1nEbw8fKnCKWDeFyBoZRbgl hTriINjWLAAiB5hokEUF9zVCOnV5brg= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Q/8wTtFHWOY8WCxaU8HkI/yiAQudzVdJHBFV92sa7o6ihCOVwUO5XKyepSUB+C3knibSpiVjAqxAVJiYI6yNfZUBV10noa4uFp0j1QPyxi4otj71AFakrs6uNKB8jeOH/A08m08ha044KccT+lEgMdOZAFS9CQSdsY172VTCrqTRIy8YTb/QwplQT7PpaWwe4Rpy3oty/XyiSj5dFLmVC47Tj2onFYBHECgwjfRhA/4KmnXFctl91kvP3hypsE905LjaSMIT1T370TGhPqRqqB8Ca59Ovi6m2A515u68VXQyWFG4NhUoRJSdVH+hlpGtkvQiPNm4YYDM5MBYhQmvKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6SMMUmKUWMrpwMdn6et7YI8Q82K0AaYpk74A+1JAtSw=; b=vFWAU1QPN9HWKNsSOfcS9+Mt0xjgviQGqT+tfDTR/k0tJQjiiBH5vv9SHiyZv8Q0ztx4BCwj126pO9vm1YrlZCgqWjH6cVKxuNo+1/5aVDYOrfkJsht2o0KS8lssxMh4xouwTKgrOQNquUi0al3RLOheA6qZM8WBalIsPeA+LulcKpWI0AOC6vAu95GkyaeglrCRZOL7R5IKZmBQvKX0XyhscjUtzCVuP9aiqhfafIkDrf/ho/CABs1RvSdu5lrO3nm1S88ToJiIvC6yhHAA3vNSbvEl51SYEJlMDq45e3/V6IiFfX3q7e/bpRWo/UJ7grtjshNwJarN0AGIP9/s1A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6SMMUmKUWMrpwMdn6et7YI8Q82K0AaYpk74A+1JAtSw=; b=hNSwAwKWBd5TeZ2KvbFE69hCEYqD6GAf+n/20zYRaQkxdq+j7nTPcrNnPEofzDdjmouLVI1xDk5ZQiXkruwwq0mXTSvXTY6AZd25LSa4ZIDIcHDweqd9epu0yN+OJrClfy89Gf3WWv2RIqHyxqj+NsNRiJvtdzWn1NCPla/RGjIF9zrSD7ihUhYgGU6WY8LMuInnoUyWU4z4+kDGiIvLeJ8fequFu+jXmNkIkWtL168F/eMeiEhS3P5ZerroaM5PpndEskTT34FCU45HaJ5B8JlkKoSjqDEUFNWJG8YTMGgs8z4UNTv+GIMHUqWTB7QZgS68hB+u3WvYLoL8jQSPow== Received: from SJ0PR12MB5469.namprd12.prod.outlook.com (2603:10b6:a03:37f::16) by MN2PR12MB4048.namprd12.prod.outlook.com (2603:10b6:208:1d5::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8114.31; Tue, 5 Nov 2024 03:29:47 +0000 Received: from SJ0PR12MB5469.namprd12.prod.outlook.com ([fe80::ff21:d180:55f2:d0c0]) by SJ0PR12MB5469.namprd12.prod.outlook.com ([fe80::ff21:d180:55f2:d0c0%4]) with mapi id 15.20.8114.028; Tue, 5 Nov 2024 03:29:47 +0000 From: John Hubbard To: Andrew Morton Cc: LKML , linux-mm@kvack.org, John Hubbard , David Hildenbrand , Vivek Kasireddy , Dave Airlie , Gerd Hoffmann , Matthew Wilcox , Christoph Hellwig , Jason Gunthorpe , Peter Xu , Arnd Bergmann , Daniel Vetter , Dongwon Kim , Hugh Dickins , Junxiao Chang , Mike Kravetz , Oscar Salvador , linux-stable@vger.kernel.org Subject: [PATCH v2 1/1] [PATCH] mm/gup: avoid an unnecessary allocation call for FOLL_LONGTERM cases Date: Mon, 4 Nov 2024 19:29:44 -0800 Message-ID: <20241105032944.141488-2-jhubbard@nvidia.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241105032944.141488-1-jhubbard@nvidia.com> References: <20241105032944.141488-1-jhubbard@nvidia.com> X-NVConfidentiality: public X-ClientProxiedBy: SJ2PR07CA0010.namprd07.prod.outlook.com (2603:10b6:a03:505::27) To SJ0PR12MB5469.namprd12.prod.outlook.com (2603:10b6:a03:37f::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ0PR12MB5469:EE_|MN2PR12MB4048:EE_ X-MS-Office365-Filtering-Correlation-Id: ecaa6393-bb27-4204-9211-08dcfd4a1950 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|376014|1800799024; X-Microsoft-Antispam-Message-Info: 73Uw5jilhYN/Z3zsyT09vu4XTejwHmU8L/L+LOAovKL9REQumkv74fXl+Y41vfWJUy8lrcWbh1Hsgpij3FUnzveNBpLb1Ce06vOvAqXGUg9+LiOvrO6J/bcKXpOblIPDodAvKk2tfl1mLuPCLZ2OgJPCzcnPkELSYxmvaJ8uO1hzAH2Ccln4GJdHZy8XP/m9ti8tOe+GIpv8mq4xPNyL0rqwmvKbnlOufaHTI39ur+KO4Mqv2zpl6cX30/Mj1S2KfO4eVRTycavMLiW0tQ7qthLaojH0fNkb2GhAWP18+CNEfXI8+uI6Goka8KIbe92WDyna/go22G6b05rnb3H2mQsJTGb+OVS5+5uQb3CgGk7L+sTQrycyt1mjtVw8GRaOniYW3ocfjLLU2Jqv7TKJvBrbmsNbzY5FGRFoboXznhtUA3pS8qLu9fZLfCN+fkihCs3ZYacovZ69mrOEjsLZBMGW5tKKj86DT/wgSSanasc266l0yAIcSxqTkcDtIw2HYzW0KXCfOUXdiwCs8qwS8YWiLrdUlmLS52EpsehVqN4xQb/7uxnveIzc9FC4/Lm809ABZrFEck49NwSxLNtXdaZtd0aeCSdfrdLOKk7lpWdKAAHPjJpOkTW+HG3n2BRxPEeUkv04WcCfRalhSY1TLRNi23IJPALKYrBZN0kUm1HG9QtpYVPbebLjIbDrkPA1zGceBVYyLzjkKYaaBSR218c0+LkTolszkcWJ80NMUzUsFlaCYPIoZtzUgUJ9xLJxhU/xoLaud9hcnn/8Bksh03T3C43AwPyh2mHyb7uqU8kEoNElnuLz3RRk+brjQ/x1joL7uhznMvOa0fTcSKRQWpy10tgI35xdBtsD1LXARxh60jX0fYIzggqUoHcAar5Ut8mbM2ItInq/pWVfKiTpqsowvGfgybEo0IL9QvW+kZNjPtBD4j5DdpWkxLiH3/pd4D/KTC/PIdwZVsaTUFkCSNLrBviAbwhbacJUpWj6HyQwVLu8PvXFoy1tWI+BQ9ev0lNVrJoWcShtuxf0H+Ac2dVTMIPS8e5OOgMBj2+fPaMQL7s9agNu02Zj+KcFvRhAdxipHpA+OiH3LmBnmwoOVqsPNcbak/xGXuc7ZKHizZdSj/410mYl75xWkzgIF85FdsBsv2lG9R+80uIJ09ukseJ2jot386+rD7B2e9L3MGaxe3r/XSFAF/QRIcqF+7IbfZTLKzYApU8EONPbuqiyyjWrbJKGXMbo08IKV42dez4k/XMGpEb+KkxST3tJSd/Vinm76etbdq0cX2YzUOcjptZQlbZ1kvZ4gmWdD19MTwSCHEoaFVffBOV3xyx++Rn/ X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SJ0PR12MB5469.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(7416014)(376014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: J8luCd9GMSV4XAjPAObcubNnnNyxHh7n4OO96ttca9mpopLxCAGS4oWq9i5Yt5pO6OCOL76HD3vpvQXpErhZz5Q6/FszqBLi64I+3FIzryDMZOLBAMTrU3cUMoteUReOo8hrhg2RrUhgrdBek+Arw1exi/h7243R/xxtLHb5jbbrDsqzMLSoAKL8GZXwHbSVrtiZI00f1ogzCV0q6WXaJxAcbT1e7zHFaPwWwAq3gN36bR8TlWwguil/pzqOgWu18K00zhwzB/wk/Nk8YyKdt2fyhwgCytaas7OI3T5ZX4G0AvEe+ZXu1xRl4PbzAWsBZ3t3T1ltWpshVv3q6rBYTjHvlEo8XdG474G9Mk1cG6E8aCr7BqmDSdfZf5EiyV7DEXlQnavqgxmF7dULf2PKhN7Wp9eSvQdl1Kiw1OGtjDITRHLeaiBHmCqmelqxLZ5NlUmh4mspmhgLEfJ6EiVAY1XhzEtCi99rzxDNc8dlhloG5DE+U2X+tx4BCi+uDkxjfrkw4Ak+PQdR2HwVni/AzcAVr0iYi52Oq+Wov/KXS9Ma+3psSBIflE8FPnOV7L38dkV8mcUrl05FS/oTXg+ITGDO1ZNXYQDsbLmAXxnyVGviMgNsLJTbK8M+OCa4p2ttbVqRzSoQhs2HGbeWOY24z0iuLgetGdDcQY54vcqgPWQqtF4OU3moLHh19HOkWwfN8kvj6NJrKXT966kYe/nPsLsfTcFb9861q/UwHp52LULBOpb5HX+77o8SjIi7Yg3onhpeGZnOtuF1PBTNpt66icFj0dFLJuN9nhe98fAd+E4Hq98wBJ4Ubi8IzRHhFXsziDTtnIMxitep6AKn50rMjZcGS43CXA2m/FN8yezkrBpw+lPeIaxIaQ7VEXXrcSUqptHEDjj2VgNwUr5KAYWqmqLV9eso+u+CJtum4VwJjKaePPzO18diWTAzi+jdEmR+qQtffmK4H7D7CPSG2dQPiPZSU0Y2pectTAIWTXqIPKVhm12Ir2VcUYr7wfZYzDEDawz7n/aX+w0TvCW0MKau+krIRTIFY/Hz1cM0Nnwj9BdY259YF2GtU1JlB+aM5BdOX+31QK4fPorr2sFyIrqajzcUiDK5onWbSOdkQ717+a7p8m0hO083rNEf4HtILgmH1r12/lM8xJ566I9nED3FP2Sqffz4eKEd+dff+VS1Z5KESo1oY6alsnel/fSP1y28zRc6z2P7tjuVfyx/eLEeLtnAqPwW5NGZ112Bg/epVoYntPWDj67m/+Z94VnlJ8iZslrwyN9FD8lO3vmY72oyTpuRk8r3x76cg/CzcJ2TWErwkFp4vP3C8y9v0zT24MZ/wY5WYzHJNtmOQAbhp8VbtvYk3O7gKb50iAdyKTwci6VpnyV2JGECNCS5cwtv8clHiZsjIBgQpnyGUiLPnEwxI7UZ7eM0cVQEe6NV0TbUQPN/WUm6CTsVAoU97WlJbxyKldA0hbKMota+wv1NNryCLshuz5CRhz/AWRFk6W99zzTEMZ4lVLY3hHnTfL/xo/NMKPCkVsQx/4VOzynQ169NdGML0ZFjsFF/+X0YcOmjmJqt3U5rpTGBE1QqlqWV20mq X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: ecaa6393-bb27-4204-9211-08dcfd4a1950 X-MS-Exchange-CrossTenant-AuthSource: SJ0PR12MB5469.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Nov 2024 03:29:47.4736 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: xCUQTBHC0mJpI5AoBEAJSKhTxpg6avqGFm7SS/yRuLmw3mAvtia35KSsaxn22ISXE5pshLxDhqnORK7fupfEIQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4048 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 55018180021 X-Stat-Signature: yf1itrzddmesmw4zc9w7nj4xhrpr3kpg X-Rspam-User: X-HE-Tag: 1730777382-547372 X-HE-Meta: U2FsdGVkX1+PPoDBskt6xdLkJkBLlT7QNsjzoy87+0SzNI8EdNJY0e+xHmDuXJPItFe7/mUFcGvRXSPg/LDj6UpFtgXfqlCFK1RqkEXF7mVch/52wSITN/ns88D/G+52hD6nPOFiNhKuaM5oaoSUjfoAlSCdGcwn96b9y6N93lPuEbLGECbM0jfYpPTDzHrT4aAL0ZwEnB1+BWn+/P7D/nICDpQsLiwVXyXZUN2DdCGF9flK5poG5V6w74ku30jMKTZvTj6R5SW6xYY/C4z7q1IsLQnbDvmZwf6byw26DkyUcdCrXOVNYr9m1pJ/F+IcVq3lmfBa6NUeYFSQnk7y5HveUIRyPz6aF84mb3MEhkCjlh+ZUf5SdZau87gejJr/c+kOIr6pw7whRa4qoLNsJDLhfxqDLlyZ6hOvw4TtXCTjVTA4RB41YhbHjqYwNb3cKr8/nUEcCTGvFaZZhZnm0VZNGhex8pRhuO0TKpo6RB8xau6PwmGOcIS89vYiX9app+CuABlI8R6FdE0hyQZqDUPX6rhPn6uTgzs4k1D90p84v3LgVO/4CNfYo4J+sG6XzG/5xyIbkc0zX0zKnC3kJuLQkVYCqAyhuQ6gKy+c+239z4qEV2HRRa9uZjnSCiaPSw2zQbwpRyINENgcMhx1GAMtT/E8TCLKRVBOBRW49t2oT7LYEJhP8swlb1I3b2DNMu0ZKGTUuZTDO/ea/PEi2lI/XeW9Mpe7uDjJiFchmIdDZsG/bV2Oo4B2C/E5BHPYQsBilirn33oCUDluGkiizAZvGBRNl3cq+rxRKXcTdmj9MaiOaA3a65yy4Y758u3cfjsrv++msRFbwu0hsOtd+GHnUDTSu/BJN+mA9MBvpvJMiYyfkrgScBGkJdE7OkEDiQwoFLaIWWiIDBmjpCdQZmDaa5Eue4d8COdtc9+JMaOBUBQO4PpkSVT+ayMopbS/CNPamEpiLKsui9VoDZX 5jIHtzFe e8jsVXg+vtD/GJxC3mhLL71BvESAw/XPyCPYVSDPP6CQubD86D5ElpOZFyWUONIos6eg8Pcu03ZWm6y8OR+1JOv4csgnzfb6KrSbTQ9mG+BZ9ElBTKmtHuMlXgitgspYM4xmPwcE57+KkMwpOzhCWn4lsI84kV2PJ+04+kkHQchopBtBm6MdM/622u6URo1tV9vmmdmNbJ+SOpOp90tyLIN2QloBRV5KJn+FpbiQmMSrWrDtwtfQUv0DW7I1tfxfZXOmA/QPqtFKsXFcs6IFr4jgqiUAqyLHHRCQlkhvqdVUCud6gON9KKZwY7pTF2MJcBas0zk4zFWQOOoQ7aEXunWcGacoaIMiMOhiK0l8GPf376t8/o+rfq+prk05XyLS30KHAm6Sv728YPamYRixuOMKZW/A2VjOJ2Z9HaOCH+wMPFmP77OmDFpygXUpT9vOHNBke0ILumdTeltR9QqqPcPY7syVoRW2981Gd/3wRuBUa39u849qNcsOX/wB3Zqj7ceBpZj7tkdVWXrA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: commit 53ba78de064b ("mm/gup: introduce check_and_migrate_movable_folios()") created a new constraint on the pin_user_pages*() API family: a potentially large internal allocation must now occur, for FOLL_LONGTERM cases. A user-visible consequence has now appeared: user space can no longer pin more than 2GB of memory anymore on x86_64. That's because, on a 4KB PAGE_SIZE system, when user space tries to (indirectly, via a device driver that calls pin_user_pages()) pin 2GB, this requires an allocation of a folio pointers array of MAX_PAGE_ORDER size, which is the limit for kmalloc(). In addition to the directly visible effect described above, there is also the problem of adding an unnecessary allocation. The **pages array argument has already been allocated, and there is no need for a redundant **folios array allocation in this case. Fix this by avoiding the new allocation entirely. This is done by referring to either the original page[i] within **pages, or to the associated folio. Thanks to David Hildenbrand for suggesting this approach and for providing the initial implementation (which I've tested and adjusted slightly) as well. Fixes: 53ba78de064b ("mm/gup: introduce check_and_migrate_movable_folios()") Suggested-by: David Hildenbrand Cc: Vivek Kasireddy Cc: Dave Airlie Cc: Gerd Hoffmann Cc: Matthew Wilcox Cc: Christoph Hellwig Cc: Jason Gunthorpe Cc: Peter Xu Cc: Arnd Bergmann Cc: Daniel Vetter Cc: Dongwon Kim Cc: Hugh Dickins Cc: Junxiao Chang Cc: Mike Kravetz Cc: Oscar Salvador Cc: linux-stable@vger.kernel.org Signed-off-by: John Hubbard --- mm/gup.c | 116 ++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 77 insertions(+), 39 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 4637dab7b54f..0f5121ad0b9f 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2273,20 +2273,57 @@ struct page *get_dump_page(unsigned long addr) #endif /* CONFIG_ELF_CORE */ #ifdef CONFIG_MIGRATION + +/* + * An array of either pages or folios ("pofs"). Although it may seem tempting to + * avoid this complication, by simply interpreting a list of folios as a list of + * pages, that approach won't work in the longer term, because eventually the + * layouts of struct page and struct folio will become completely different. + * Furthermore, this pof approach avoids excessive page_folio() calls. + */ +struct pages_or_folios { + union { + struct page **pages; + struct folio **folios; + void **entries; + }; + bool has_folios; + long nr_entries; +}; + +static struct folio *pofs_get_folio(struct pages_or_folios *pofs, long i) +{ + if (pofs->has_folios) + return pofs->folios[i]; + return page_folio(pofs->pages[i]); +} + +static void pofs_clear_entry(struct pages_or_folios *pofs, long i) +{ + pofs->entries[i] = NULL; +} + +static void pofs_unpin(struct pages_or_folios *pofs) +{ + if (pofs->has_folios) + unpin_folios(pofs->folios, pofs->nr_entries); + else + unpin_user_pages(pofs->pages, pofs->nr_entries); +} + /* * Returns the number of collected folios. Return value is always >= 0. */ static unsigned long collect_longterm_unpinnable_folios( - struct list_head *movable_folio_list, - unsigned long nr_folios, - struct folio **folios) + struct list_head *movable_folio_list, + struct pages_or_folios *pofs) { unsigned long i, collected = 0; struct folio *prev_folio = NULL; bool drain_allow = true; - for (i = 0; i < nr_folios; i++) { - struct folio *folio = folios[i]; + for (i = 0; i < pofs->nr_entries; i++) { + struct folio *folio = pofs_get_folio(pofs, i); if (folio == prev_folio) continue; @@ -2327,16 +2364,15 @@ static unsigned long collect_longterm_unpinnable_folios( * Returns -EAGAIN if all folios were successfully migrated or -errno for * failure (or partial success). */ -static int migrate_longterm_unpinnable_folios( - struct list_head *movable_folio_list, - unsigned long nr_folios, - struct folio **folios) +static int +migrate_longterm_unpinnable_folios(struct list_head *movable_folio_list, + struct pages_or_folios *pofs) { int ret; unsigned long i; - for (i = 0; i < nr_folios; i++) { - struct folio *folio = folios[i]; + for (i = 0; i < pofs->nr_entries; i++) { + struct folio *folio = pofs_get_folio(pofs, i); if (folio_is_device_coherent(folio)) { /* @@ -2344,7 +2380,7 @@ static int migrate_longterm_unpinnable_folios( * convert the pin on the source folio to a normal * reference. */ - folios[i] = NULL; + pofs_clear_entry(pofs, i); folio_get(folio); gup_put_folio(folio, 1, FOLL_PIN); @@ -2363,8 +2399,8 @@ static int migrate_longterm_unpinnable_folios( * calling folio_isolate_lru() which takes a reference so the * folio won't be freed if it's migrating. */ - unpin_folio(folios[i]); - folios[i] = NULL; + unpin_folio(pofs_get_folio(pofs, i)); + pofs_clear_entry(pofs, i); } if (!list_empty(movable_folio_list)) { @@ -2387,12 +2423,26 @@ static int migrate_longterm_unpinnable_folios( return -EAGAIN; err: - unpin_folios(folios, nr_folios); + pofs_unpin(pofs); putback_movable_pages(movable_folio_list); return ret; } +static long +check_and_migrate_movable_pages_or_folios(struct pages_or_folios *pofs) +{ + LIST_HEAD(movable_folio_list); + unsigned long collected; + + collected = + collect_longterm_unpinnable_folios(&movable_folio_list, pofs); + if (!collected) + return 0; + + return migrate_longterm_unpinnable_folios(&movable_folio_list, pofs); +} + /* * Check whether all folios are *allowed* to be pinned indefinitely (long term). * Rather confusingly, all folios in the range are required to be pinned via @@ -2417,16 +2467,13 @@ static int migrate_longterm_unpinnable_folios( static long check_and_migrate_movable_folios(unsigned long nr_folios, struct folio **folios) { - unsigned long collected; - LIST_HEAD(movable_folio_list); + struct pages_or_folios pofs = { + .folios = folios, + .has_folios = true, + .nr_entries = nr_folios, + }; - collected = collect_longterm_unpinnable_folios(&movable_folio_list, - nr_folios, folios); - if (!collected) - return 0; - - return migrate_longterm_unpinnable_folios(&movable_folio_list, - nr_folios, folios); + return check_and_migrate_movable_pages_or_folios(&pofs); } /* @@ -2436,22 +2483,13 @@ static long check_and_migrate_movable_folios(unsigned long nr_folios, static long check_and_migrate_movable_pages(unsigned long nr_pages, struct page **pages) { - struct folio **folios; - long i, ret; + struct pages_or_folios pofs = { + .pages = pages, + .has_folios = false, + .nr_entries = nr_pages, + }; - folios = kmalloc_array(nr_pages, sizeof(*folios), GFP_KERNEL); - if (!folios) { - unpin_user_pages(pages, nr_pages); - return -ENOMEM; - } - - for (i = 0; i < nr_pages; i++) - folios[i] = page_folio(pages[i]); - - ret = check_and_migrate_movable_folios(nr_pages, folios); - - kfree(folios); - return ret; + return check_and_migrate_movable_pages_or_folios(&pofs); } #else static long check_and_migrate_movable_pages(unsigned long nr_pages,