From patchwork Thu Mar 6 10:33:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Stoakes X-Patchwork-Id: 14004148 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17EFEC28B23 for ; Thu, 6 Mar 2025 10:34:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7A326280004; Thu, 6 Mar 2025 05:34:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6DC25280003; Thu, 6 Mar 2025 05:34:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 507E6280004; Thu, 6 Mar 2025 05:34:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 28898280001 for ; Thu, 6 Mar 2025 05:34:20 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 47A4056B92 for ; Thu, 6 Mar 2025 10:34:20 +0000 (UTC) X-FDA: 83190766680.19.52B07E3 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf02.hostedemail.com (Postfix) with ESMTP id DC7ED80002 for ; Thu, 6 Mar 2025 10:34:16 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=ZSnjWGmQ; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=cCpAFuMc; dmarc=pass (policy=reject) header.from=oracle.com; spf=pass (imf02.hostedemail.com: domain of lorenzo.stoakes@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=lorenzo.stoakes@oracle.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741257257; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cTTb8e9qNXZki0winp9zDuBW42NjoXUdms+Tbpbt81A=; b=vLbmWgJq9faLz/JLnCZ+wKw+PmHI89dCpSnNVnhvLG/5TA5khyFBS+Q5wVZj8LpDnZCcGa R14BpiPh8iD6A6H1TwOfhXnoYGS6fvKpG82tq7aDW05lCRqxmlNYznETGQD/5Sc0afUUvo xZ3QXOw2AzDMiOr5HAc45x2T3AUDCdU= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1741257257; a=rsa-sha256; cv=pass; b=38UJV+w6yQveQb4YrzYAz8RR1cUHhHan8vk+th6iN99+xrij3x7w185f6MDk4hPYZAic5z DkQQ1qOi9vOQ24YE6zHrkKI6ECJcp4InN+ypFUCRxQyTT0rSN73C38HHVqshyjW/aL+XEY c3e+tX0hfPTbppmT81nA3oQ4KQ186G4= ARC-Authentication-Results: i=2; imf02.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=ZSnjWGmQ; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=cCpAFuMc; dmarc=pass (policy=reject) header.from=oracle.com; spf=pass (imf02.hostedemail.com: domain of lorenzo.stoakes@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=lorenzo.stoakes@oracle.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 526ABj7T022611; Thu, 6 Mar 2025 10:34:13 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=cTTb8e9qNXZki0winp9zDuBW42NjoXUdms+Tbpbt81A=; b= ZSnjWGmQU+48kCfE3ljhmhRtZZ7sBdIMt47/Qpnbm7zYYrlYhsCnHO4eeuSdMdzj 3bHyCCTjRwi1phKlUxFIJog0OYOk51vPa9a58LXu92kss2G0iZ9YP07COodrbZOJ dHzod6t0UVI/4YLkhXAFVNloi0G1l3pSc5JtlQUOJ34mnxybPIenvugrM+4qRF+b 9pzBdc7QFScww/qsetJRtkPPFQ1KZPuKlBDHq86z1oCyr/IPCsYEFA2ftFuvDw4P i8RYpH3SznPZr3JZ1288a8EzM16as9cPHcLgcgySVXoAnrNu95pizsyD8giMWyZP rjap1zKFr4woa3Ki7tFUJA== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 453uaw1sa4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 06 Mar 2025 10:34:13 +0000 (GMT) Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 5269KCmp015762; Thu, 6 Mar 2025 10:34:12 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2171.outbound.protection.outlook.com [104.47.56.171]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 453rpd0r16-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 06 Mar 2025 10:34:12 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=c6h+XU9dXv7K4vmC3XghuVkv6VVH93fToKLVFRxaZ8eQIW5YdybI3t7cKkR5fmzsZHfLq4H/mKWV//+pubjR7RCD3QPjdFWGfEMJM0abXhAeLZ2zNrs4sWHDaIa4tLIm05FT20vpysFxkyZ5qq33HE+FN0/Pxwdn9r+szejZKtrqPiUqLQLsEHN0msNxV7FDrDEx3zJwdkp6KKNBKlSDgHdb2DzDXodk8MoMxqKEue/T3Vx1oThJj4ssBojEbj9UuLLnPZAmqxmCJzbaePuLyPFtAxlr8ZMkMxv9F8Sb1qJCLYZuhCZJhUQDaIGbEfv/YyCA3RIkS5ryLfDrPaQx8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cTTb8e9qNXZki0winp9zDuBW42NjoXUdms+Tbpbt81A=; b=QsN7xI9uEVt/miCW7lIwINgEE+2iapwdoGzhligVvt+W4352r8HyGV1OVpUCg/1XGHJGKXON236vp9xrJNI+cwVMS3Na+KzqJ1ENPw9g4LpSCBWKWzDvPT9EVbn+5v81v541vVd6ZkZQHqdU96Lp4sR9aYO6mkCnZzC1/GCqm/nR1InJGEKIlhEk+sDNFftdAMnnAC47pfQvLQqghnnefRYkjuNkvE9sG1GPrJdI95azibOToXsKtg7L7s3sDEqldvbQpLqYZnAR2LEpYG3v2dR3dEGTaC6AQzrdOB7SjUk0sQ/2Lqx2HRVK974UNKw/9eT7YRKGpG2dMWqkNmVP6A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cTTb8e9qNXZki0winp9zDuBW42NjoXUdms+Tbpbt81A=; b=cCpAFuMcNafLKUew/BkUKy4gOcidOnvykk/+QG65e4DN1aVxyfuKXqkuNqyUoa8bLE4zdmhNvKlmRvYJoawNe7/NkjOQW8nNB4J7DE1o2lrc7n73+ayhziqyrXhx8Qh/bnaPP5FScG1RdcGoOjY9w5G7jDPMglPw8Xl2eSligEE= Received: from MN2PR10MB4112.namprd10.prod.outlook.com (2603:10b6:208:11e::33) by PH0PR10MB4647.namprd10.prod.outlook.com (2603:10b6:510:43::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.20; Thu, 6 Mar 2025 10:34:10 +0000 Received: from MN2PR10MB4112.namprd10.prod.outlook.com ([fe80::3256:3c8c:73a9:5b9c]) by MN2PR10MB4112.namprd10.prod.outlook.com ([fe80::3256:3c8c:73a9:5b9c%7]) with mapi id 15.20.8489.025; Thu, 6 Mar 2025 10:34:10 +0000 From: Lorenzo Stoakes To: Andrew Morton Cc: "Liam R . Howlett" , Vlastimil Babka , Jann Horn , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Harry Yoo , Yosry Ahmed Subject: [PATCH v2 1/7] mm/mremap: correctly handle partial mremap() of VMA starting at 0 Date: Thu, 6 Mar 2025 10:33:57 +0000 Message-ID: <94681428faba4c34a76e2de1c875629372aae3d5.1741256580.git.lorenzo.stoakes@oracle.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: X-ClientProxiedBy: LO4P265CA0164.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:312::13) To MN2PR10MB4112.namprd10.prod.outlook.com (2603:10b6:208:11e::33) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN2PR10MB4112:EE_|PH0PR10MB4647:EE_ X-MS-Office365-Filtering-Correlation-Id: 3be15332-f440-4e78-f969-08dd5c9a6e19 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|366016; X-Microsoft-Antispam-Message-Info: QPv/2vshz+GfhcSUDpKs0yQnkIhupwEvCZd7jdbXM8394liO0TlkQ5OGkLxihzj/W4yBYTSDNBxzbphYpGtHGvicEbb3AZ5dkGtf9uffaFEaYMz6V//W0WJ+DIzqqvQIsfjb3vxnmT1NDiB4818vxoUU0mhyWD1McSSghGP0H/jTow6q503NqqVsWdUyUFeZTOugKrGmT+bnLosC72cZn2vJVcmLKjOiTyERHtUaCbMg8iTNRknEhwujDxMdeqML6m3+qJpwbVbTTtah2YH62ZDpKAivnQLgXg1Qev5UlOiT1Frx8OnVrQm4nefqE+xwcfA0wKju1K5Q+Z2sLiWPPOAaQD5ykFBhRAcVIPvw/RTZSgtt4eiml7NUFhZb+Nefrk+V5257/jbWEADmriIOzCVbaFaqZ9k9Y1Ou0HEfbdpN35UicBZoW3LKLsKQbvwMb2Azq1dQs1F2HShClr/zBXMbzivtYI8Ns81Sq1Nlez92GK+GA7JUpP+QLO4SISZmxKz4LcyCmGAAGAOSg05F9ss438wuCiCWtDt7UCLlFYPqFG5wNtodM92Pr5J88qFdNeJHYAoM1RSYK2upFFTj9/3WuBm66MBiosqMnO91tGarmpGN9Lon41m5PZOoqTyVRJqXwh+TemupKc6vEnMiKQljP4OFKdzYWoRp9qs8YDZsz/xFH2BQYuo4F9uomGwoX6/vz6D+B33SNJ6TLxSEsxN6ZxHIkhdeZ+ogVgHRew6dRHFaTUVUBfyAt4GdSX4oTUgw31bvjg6uLxYivIjj4K2Yj14xzSdrnnzqwYlRv33s+Ec2Bh+r8togD2BtAU3hswD005REfCW1VVUgSgTMyNQszd3JfWWNNQy+4rhU0LfZjhw9mBptC7L8Y+lfaovSBTQAc47CzonvCEqLJf92YIr/7WkGgRAurjvza5jC0L/nb/PNJId8uRyWCcnx46vkl53KqzhCZYkDzL2m0OlAV/KhOTzDCPS3TiwMVLUBwHYDoEcqPz4qEYRN0bJH9InpEHMQRLhrIhaSdjFk1H5G+XIinquqlNL2JPQEeLh5f3gBSzvmzpS0RvPO6GfWVAX9ddnfB6Aqyy1OziJu++qWCV0eE26yNNVsf4G2y78Bk3hBL0tu2kt/neEdquVmF0V0pT4O4+FKg/bst6L5Qltigu92R66pYzNzwlBfnh71cV8UQan/MltxUWO+6JUoY4wnVx4wm+XTIb+6/hH2oHIUA28Fd3mPdMkTRfb5d3q4zRWpFFaC8f9EFmtNvl9ieKozyN7As/izV9mWMss0doYf/Gyq+MzOHqCgjV5NJS20QduZqQxGjmXiD8HBJ5e9XxuhBzDuFSiQM3JrvlXx7mhNdC+0hxMwAt4NOl0wh1qtCq4JH2mq/lTei1RNAdgIA1JY X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MN2PR10MB4112.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 1oA7oi61/1PDxL/QDrNe12KqYvWNs0jc2eBuAf4PytOPnXeLDHMEehcpKC9Nd2W2C9nMQzYs7F20j9nW6UXTiGVVWVNFBruAJBHlfqpgdumadiZFSpCT8FqvnxNViQl4tKVGTSpp+d7BnquYPKqsbMy8ZWRl6eGE4M3/SO+aSnLm/RAvCn0TyXSh98xpspr+guIcG3bBsAvelweJEK2XeYABkrgDF4fgnahQwTUJmZfw2LKqGALf73og7kRSnKrvN0WWRUUcobG9DUtR13W2/HuBwUtyssgK2GpouoGdWtCn8RjXPdez2GOzmJjyr75Wo08OPZpZx7dnC3802aCEcJLLsnA9Be+tTeUQihiy9oKMWLNCjtSq2JqH9RGcP7fkH2mwv3A3+HgfAYF8P5nioHGdjSmXXUJXsiW5k/HMR6m+TM2XOEF08h6HhK6v3KJEkM/dZ2a3fCcw+gtt8sFBCPlZhpAMFHxd58sOpVRNj05CX6bz0HMAB9Y28BPC6SmRa1RheiNUfOzJlUXy1L5DfQdX53ntXKdZl2Cv7WB6koNuEfECkALxXKc/bYRgCOt81fknP1jaJvVe/tT3I31wMX/TEgHeXoisS5kkHCc6Da5A8tNjAl4wC4DEtLEqaEYOmTb0wsS11nSRyxtV5BPAwsrjS0VjVcnnFRR/I8AAymuJDhd7UVCV1u4uZuR3kR4k6Rn/UbK0NtjiTEVs4jzg5OTonW93OOc+wRBc30cs3mOpWo/JDHenRbvjFODMnqTMj5ytcq+eysziDXYlbfhy+IgEGdwIN5HDNCLtMiU3IFJ8WNE/073IPLc9abK77j75mijal7c0Bt6tKumJnov6vfiJ5vXps7knHltGfzoJLoaDFHwd8o0aY8GYYUb8O+zmwqf0vOtnLthanVtnmBsK9NyVW//wPOgIYubQQQ4EigrcnOFoKsYuwwYPxhn+k3l1NRn1MwZwyCFW3LLLb8zKerJe5rN0XpCE/CYmUm1hiDRyc9R862tujtIMXOymkFIXY3LY7dJQHT4gJlOquNGzpLP+wYc0hbRSghlnxNKLPo5D2C9If7anEZdaPBsN+4tebtQNByGFoKuE8RWby2VWa49L0XLBkBxcXoQU1g1ire3dmDwbZZmwcIq0xTic+KybAD/XprjjrRW3XxYR3wUg0W4jnMNVk84biARXns4WT5Zcb5u+6DHpoGXRcjbMHk6PT0jCSNryfKFKEUUadePo9XHCgaRK28v7ZBotWaYO9NLEP0ZwiwqAMDNUCyqBmP9Jat5hycduQH7PU4YvU66WoCJiBEgJDSMcELXpsovsedaaSVfyjx59iix0nli/S5hLeNwsJl2XuOtZrYm5jSA+QNpoSE8LoeD5gCOetQonD6CjJt18WsBk6Wa8mGoiflkCDM8QB0slKcBYf468Zy2L1EUYsiAc15AzwP2+LqNn6RbXpgFq1kzFkSYIMeRfktRW5tXlZQR496CAzHf59frLAnxRbbuARnHINIsvhJ8H6lS+VLSoeMmETM2LVn2IyrKKJT5oYngvONDAd9LlQqYQxVSGCiGE11t3CDa4GsqpLauCzA3lyS5l22ooCHm63CDSvc0GOTM8XUUlwHoxGCanyA== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: HQo6OBAYEoWwSHrnMIAe+9GWf+yi5iVVwzd/9YMWqQaoPOMCP47F2KMzUDocfQDgnEEWRlOPpE9hLggw2+PYrhiAPf+ZDxaE6cryCRjkoOajMFVMa5aZYbp/NTRoiO4zivszXKkIasu1jHNaOe8BeaFGcNR4ZqQeMC4KKJV+rYe5kThXMV4UO3Pfr0lSc+pqWXtsDDqFmPeXjwpdS+9UffTl3AS85FQPvOkFCO2BPXLCzwUckmMaaZ21PKnfpZ4lDQS0kviaLP69kkpuMRe/s7dfiPDIbL/Y/ULFOpo+vtX793f6XITnP3QbGXxt6mwUbx7acp5YEccVlS4a1W5WhEEOyYu0VOPajFkSQHkeZYEf9iHgZ8Aw+MoLgl3eFlOp28KYaUyaopggKpyij/gujo6mC5vMK+Q3fpsbfZQhq5Jx9OApyLcENv0ILd66FsRp+7SDpP9olXKpp/GF9NoKGIuVgmmUIOBhK4UrJSEQBg04a8TV5gXucMQ++s2mX4GrUlg7zWQv8lPIGrFHbt8s1BzWaBofjFKi67bttXy4XQ4gXe3mA58F5E7eD68f3u7pFpzB5rTviyL3jyhqBpVT6eDDTAv00+kWVNrTThpyhtw= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3be15332-f440-4e78-f969-08dd5c9a6e19 X-MS-Exchange-CrossTenant-AuthSource: MN2PR10MB4112.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Mar 2025 10:34:09.9673 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: IGLSESlpAI8sW5pmuVHItO9uDRG/eQ4Cec/VwUuLDyPadPPUKm/XQnwISPs09aKG9ezh4uXO42ArgmBZXZUoo567VNR34Ez+oYNW9xsFHik= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB4647 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1093,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-03-06_05,2025-03-06_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 suspectscore=0 phishscore=0 adultscore=0 mlxlogscore=999 malwarescore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502100000 definitions=main-2503060079 X-Proofpoint-GUID: 45gcKt8moDfUzUqrHSsxlSsRMvfMSOLS X-Proofpoint-ORIG-GUID: 45gcKt8moDfUzUqrHSsxlSsRMvfMSOLS X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: DC7ED80002 X-Stat-Signature: fu3fpbyhzzfgqumx9qb6ez9wqmkgxw1f X-HE-Tag: 1741257256-441021 X-HE-Meta: U2FsdGVkX1874xSTKn2u3ll0AXSHwz61PdKp5AhfHcWKCOcUdEMa1db8ps8C0liGIK2f3sZI5UUN0AWIuJoFo6ptBDgLs/+J/hYHT0kTE3GIznN0SLrXGHe2mOwQqrHRH/uR7DtAg7lZSOJr3/vW8hIhwbGtFAnNczV7nptMV3nJXRIH7tE5MjvBI6YG4fLcGlUZFzw04voajMcHNZQepIDez3VKqPUp43NIFjc8GQ8yy1s7fBxXDkYZDyoNs6TqlWD8y/u7UNEmluetvKe0Qykj/CMXYv9WisH/j4NUIXEOHxNYtuPRSTANLRMze2vVVwlxJGTRyUo6HcvnJRmVfl7YUHliTIcd3ap/62xqLFi6XpBcF8dmibDlhjGu1wl16OXMBThGt8WJBxCdVhT76jww3W/xz33fm2HtduSzymDhGqdokZ133oR14nczfpFvowbCKT/BnxhNa1kFsykd4KyvyRcPhQltK+d2N5EOJcNM3IiKVveQRYuXvIBeRD5v2A9Kc4COBfNZGhxESmsXny3FUGjld7N5h6RUKyFY39wyiMVc/IXRjAW3CJyQJUpJbaOLfNPEQBGqEi6IHl3dy1xbOFsV1/vIR2qCglWcirJy8Vx00f2ywVc/m+IcA6ze+ZQPbDyfwl9A0A+doi5C/4xcxhTvxP7mWDe2EJ6V8VzFrdko/f/+d6OCEzoqEoK+C5p9GVV6nJRyMIj/lGyT+PEbTrYOeVTpuv6Sk0F4ZJAJdfh7NOaF3WCEs9m5lWojJL73naX7HHagdfQjRuTbM8/InZ/Mq9XN+7gfMFouRbfv5/aciuWwrkQeABnVpoMQRycOnouTm4thVfIflspXW7VML4xHkt6vQq9UHcqvC7EMqZa+CCn9wCOUfDoRgfT6FzCKMdIazCci+2RQ3vg4XJVmLGQnGl0UEj2fYJRM+gvpO4E45jv8HMiPjYKc6adplaoqr25Io3Wuf3j/CpD IuGZsp3q 2T5MiX1s0bBjhRILpAquW4OlFW4C9oJJV/qZORG57Jqj9CtjcJ91gq1g/diAXPF/QZfTaplZ2jdu1kLY4M/dULs7IP3koSwE9yCAYv/zwqHgOXz8dAOFk17bBsodnice/jSbQvfPyyh3anXrs1KsmooMAzn1H4dMJRU3VUtPTeHM5O3GxbVT/lJkiL8CXMawXjPrf1k/bXHUaQsx4uzNxjktZxsDw9Fz+Ehpj/L9J25lvrBCyjeVcaXf6R2dQ4LWbAid/MEKzHQvdOQMeNq+wxs8dBbgpXu9gTeBiy6g2+DbcRxkD196A2n9GcniAusqHxWotMVeKLzOsgHpZXpAjJ/F+tyLYDC/zxXCtvnZb0G0ZTRv2GKxdNRtotP1Ewe9tngusBiXEGQtpGi8VAJEpArGIfj6Cj0jZkAm86WYBcOS46fLlogaLokdXnXpT2AE8GR/YbgQ0c+h5UZtpSRcOyYYrohZw9tKLINCSF51RuMFthmVt2bYUnNxN5MMveIxEb3X10KS1A7DmX6NOY0/QNsU39NqpPopUgO3Mn1UwDZtsO6ML50x1hKxzFOv2IEd7gg45T0vlRpCkiuK38xjmHV+mvN6AlvQVZijrG+qjEAomxbixyQ9NjDVxIg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Consider the case of a partial mremap() (that results in a VMA split) of an accountable VMA (i.e. which has the VM_ACCOUNT flag set) whose start address is zero, with the MREMAP_MAYMOVE flag specified and a scenario where a move does in fact occur: addr end | | v v |-------------| | vma | |-------------| 0 This move is affected by unmapping the range [addr, end). In order to prevent an incorrect decrement of accounted memory which has already been determined, the mremap() code in move_vma() clears VM_ACCOUNT from the VMA prior to doing so, before reestablishing it in each of the VMAs post-split: addr end | | v v |---| |---| | A | | B | |---| |---| Commit 6b73cff239e5 ("mm: change munmap splitting order and move_vma()") changed this logic such as to determine whether there is a need to do so by establishing account_start and account_end and, in the instance where such an operation is required, assigning them to vma->vm_start and vma->vm_end. Later the code checks if the operation is required for 'A' referenced above thusly: if (account_start) { ... } However, if the VMA described above has vma->vm_start == 0, which is now assigned to account_start, this branch will not be executed. As a result, the VMA 'A' above will remain stripped of its VM_ACCOUNT flag, incorrectly. The fix is to simply convert these variables to booleans and set them as required. Fixes: 6b73cff239e5 ("mm: change munmap splitting order and move_vma()") Signed-off-by: Lorenzo Stoakes Reviewed-by: Harry Yoo Reviewed-by: Liam R. Howlett Reviewed-by: Vlastimil Babka Cc: --- mm/mremap.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) -- 2.48.1 diff --git a/mm/mremap.c b/mm/mremap.c index cff7f552f909..c3e4c86d0b8d 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -705,8 +705,8 @@ static unsigned long move_vma(struct vm_area_struct *vma, unsigned long vm_flags = vma->vm_flags; unsigned long new_pgoff; unsigned long moved_len; - unsigned long account_start = 0; - unsigned long account_end = 0; + bool account_start = false; + bool account_end = false; unsigned long hiwater_vm; int err = 0; bool need_rmap_locks; @@ -790,9 +790,9 @@ static unsigned long move_vma(struct vm_area_struct *vma, if (vm_flags & VM_ACCOUNT && !(flags & MREMAP_DONTUNMAP)) { vm_flags_clear(vma, VM_ACCOUNT); if (vma->vm_start < old_addr) - account_start = vma->vm_start; + account_start = true; if (vma->vm_end > old_addr + old_len) - account_end = vma->vm_end; + account_end = true; } /* @@ -832,7 +832,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, /* OOM: unable to split vma, just get accounts right */ if (vm_flags & VM_ACCOUNT && !(flags & MREMAP_DONTUNMAP)) vm_acct_memory(old_len >> PAGE_SHIFT); - account_start = account_end = 0; + account_start = account_end = false; } if (vm_flags & VM_LOCKED) { From patchwork Thu Mar 6 10:33:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Stoakes X-Patchwork-Id: 14004149 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DD4FC282D1 for ; Thu, 6 Mar 2025 10:34:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A2690280001; Thu, 6 Mar 2025 05:34:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A014280005; Thu, 6 Mar 2025 05:34:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 57C6B280001; Thu, 6 Mar 2025 05:34:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 29E46280003 for ; Thu, 6 Mar 2025 05:34:20 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 44039141363 for ; Thu, 6 Mar 2025 10:34:20 +0000 (UTC) X-FDA: 83190766680.22.E154FD0 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf26.hostedemail.com (Postfix) with ESMTP id E498A140007 for ; Thu, 6 Mar 2025 10:34:16 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=QWuAl1RS; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=ehRFrAWi; spf=pass (imf26.hostedemail.com: domain of lorenzo.stoakes@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=lorenzo.stoakes@oracle.com; dmarc=pass (policy=reject) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741257257; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KFsDTc2m7sSYu76rEqtlN3pEVJgvjf1GbcQX824iAmc=; b=1t+vz/zTkfVfSXRQ1MXbePXC53bfspf3miaDNyNPOBBmBEqHkb5fjkPDzZkKKKOnV8XBdA s7G6TXADLcRBaBxN0xzECMnoml5C+bzx5Vbo6I8vL3mJzMVdC7PEXNdJV6c1HO85CCrmO+ IL71ciqa+XwRXBycgRri9sT2GmeD1jM= ARC-Authentication-Results: i=2; imf26.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=QWuAl1RS; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=ehRFrAWi; spf=pass (imf26.hostedemail.com: domain of lorenzo.stoakes@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=lorenzo.stoakes@oracle.com; dmarc=pass (policy=reject) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1741257257; a=rsa-sha256; cv=pass; b=uWpySO3aYkS5waonFwtERzzEF1yVAgzHBvykWWiHyblWoWCUQSj1EQOiYgC4h2zJE/Tb75 1kAbPSMVCz4zJVYRNwPUKp4f7lKHXk/+jRlCbyU56MCSY6KFKcL3XWfbom16YhmfTAI9Bl Yyg33Udb0/QHq15A/+ZgrrzFFbU8Dog= Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 526ABfhF024533; Thu, 6 Mar 2025 10:34:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=KFsDTc2m7sSYu76rEqtlN3pEVJgvjf1GbcQX824iAmc=; b= QWuAl1RSF6BXGvGSKKipBs1Q82oEFxbyyNaaRDUDfTt+gtBDva5K8XvLUPTg//5v 0EXym8UildZjYOsJJFsFdYCh0at4YHGW8Pi9lRcs2KIoJjEcYEup1TASYZkmZGKm WRcnq7eIRYZ3iSgBw3EI2OFz+UiJfiJHMTLwQ9Z+bCtxQEMMA+01oaPz74Bplvlf gLSU6xe8IESbkHhBDWJ9wS+l0K2m3HctMw/mTIIqDZAYNnKIiZ/f86i/fVvB+jpS eMfm3v9ebO9h9kTIybX76hoNuPUkD55TIq/cKZ8jwbi99FjmQCBsMrx5UMlxNASQ qoHaDzgUdINmXfN8EkP2OQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 453u8hhttb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 06 Mar 2025 10:34:15 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 526AAEKi038403; Thu, 6 Mar 2025 10:34:14 GMT Received: from nam04-bn8-obe.outbound.protection.outlook.com (mail-bn8nam04lp2042.outbound.protection.outlook.com [104.47.74.42]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 453rpj67u6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 06 Mar 2025 10:34:14 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ce7QyQj4x1DsVfbWle+9dIHDlaDZMIq9fTMLcs8ethVs5WkfDeGC/PEROyqL3pey3lLaM5Nr0YakciJ099IiJE7ndheRR79Z4YOdxTfUjCGivJk2cChp7mm5LT/XKpmHnvW2TwUBE3GgP9OhGyCvecNBSzwSNmjhL1hf9iP//M/+puGkw1pniJjrq6nFDamOn0dOEX/dhPUlrE5gPGrSk6V5LSUJrRd/wCIoPnOsBI8mr6pzIHMAlwVCYWqH9Iqliq6nSLALW4WzD9zu5Sc7vUH8Dz9nL+pQUapsO/Q49+TeJDdPy1j/4tmtsQu+4a5YpXbs8PKWQsu03aLlBP3xjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KFsDTc2m7sSYu76rEqtlN3pEVJgvjf1GbcQX824iAmc=; b=lRjQsTnwnM9AGrvZIIJNWEJxUIPqP7jfxMi2Q7UnL1KXs2HHm0ZYnH33mOtQdT7vBax/7BkETqD7d/EugVRDq9voQ7R3B3MDpZ2sFrj94ZnnFNmVyrDJDBWKGW96t1uRS20sXl7pIV2Gw+pgWJ+CXC3lMHaICg7rYjGVHemtLEpw2VkXDtcYtXvuGclGRBN6vJkDjh9jzpt0k75tZHYEEGvw4ouQKzfTopsDPnYsav3ZCkygiGUgpp8vAq8Yg5tPqC96F/duDbhU19yIEqZyiNA4fZLz2I6hs2PfkOb7it1Aun5I9LL7nmp/TTMenNdJD+w65uJrLnL48HE9Q9AqJg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KFsDTc2m7sSYu76rEqtlN3pEVJgvjf1GbcQX824iAmc=; b=ehRFrAWivrAssvSNdmh4IUNP3jaLGNXKHTMbYUFsVBsdy0HLsI1O6IvzzHWCxYScTQ+bugS72WFWiuyD8As/InXrloxDt7u1BfumTEYefA8gxrKY066oBGSzoJqDN7uFNk+r6sOAsIPEStHzOePm+k2fKsE9frfz+eT8YGgOm7k= Received: from MN2PR10MB4112.namprd10.prod.outlook.com (2603:10b6:208:11e::33) by PH0PR10MB4647.namprd10.prod.outlook.com (2603:10b6:510:43::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.20; Thu, 6 Mar 2025 10:34:12 +0000 Received: from MN2PR10MB4112.namprd10.prod.outlook.com ([fe80::3256:3c8c:73a9:5b9c]) by MN2PR10MB4112.namprd10.prod.outlook.com ([fe80::3256:3c8c:73a9:5b9c%7]) with mapi id 15.20.8489.025; Thu, 6 Mar 2025 10:34:12 +0000 From: Lorenzo Stoakes To: Andrew Morton Cc: "Liam R . Howlett" , Vlastimil Babka , Jann Horn , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Harry Yoo , Yosry Ahmed Subject: [PATCH v2 2/7] mm/mremap: refactor mremap() system call implementation Date: Thu, 6 Mar 2025 10:33:58 +0000 Message-ID: <6a62c1a4ea7f362b0685bbe3a2e74a47d24e5c43.1741256580.git.lorenzo.stoakes@oracle.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: X-ClientProxiedBy: LO4P123CA0197.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a4::22) To MN2PR10MB4112.namprd10.prod.outlook.com (2603:10b6:208:11e::33) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN2PR10MB4112:EE_|PH0PR10MB4647:EE_ X-MS-Office365-Filtering-Correlation-Id: 7e834499-ff8f-4cfd-69ef-08dd5c9a6f43 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|366016; X-Microsoft-Antispam-Message-Info: pnUmkEw1zXuGS3pciXi6o94tounfY0H/SFNkbkLnYpEaLzd4UuNQxRVwihlRqQPMJU4g2z4+pQg/Rmz3EQD6zGOJ1RxqjiYx7yrt4eRxViLv9oBrCj5MY4+7c5QbEZarR4vSt9Gk0u6/w6XmaHXM17qorVKSj/YLqXqk9kr7GoMcgnmbaDjMhIAMy4YsQgz6J4DbOAWsWwkONKXT2FGwyPYq675nK34b1RDZ2GVyeLCgtk4kd93QV10VBK9WAoYvuOGFMjqoH3S95ecTaeHbh9TbFffXFvDHGSiCc7qIO/7+9XrGmAPCcE1lCkGgGvF1y8IN3PlmPQ7hFWKQIHDghwRqE4wkB0cJtlWzEXdEqSl75PjUF1qoWFIe0WkQ2eU1p2n4jyCNwFelWhxD/KCS8aX4twPZYSBwQ4UcRKbQc4pJXEx0oBMnbdXZqp27w3P8e4WPPqp1jnTK5gq9ObhbDgRgWH/G1kCnb7QJncH8pZ3YGS6E91tWk2DAVtJvsx21Mic1gHUQTAgCaONuKvWsnNeqOHLGyBuBxT6cvNWOpH4F00RXeqB+6MrYlFJz/ZerQHijSxXV674bxFc5gR4JREHGz9yPJOPVO97AF5Po4DG1zEV3PcdzZSGcAUm41E4rvHMQczdG3nVcIqF406KcTeBg6StVWivueLPOQlARFxqX/WTbSNejxfpwsQAygX+atgGlwZbO3qrHwZFXLokk+hRtFqS2O1gBNsdL/bed3AlV2Ia/ZWrIioGSqC2h87KqAs31l9vZhWD/fO0l5bSmG3XGQiy+rEvvcCRe8A9Z8PIxvjuVb0V5qwjbufXsDJlUbZMKAKu5dhXAgC2qyZ83CBc0eMbWco8Zl7uC+VD9qyOCvonUsgPrInXqbXXIa26rEC1el8hUU4/zwJQSF0jFWP7sXO5l4TeT2Ih/yhByjgW2ZdeZgkixSBN1o+PUXWOfJXdWfJHr+2mJxlJejt3qyNxrQqjGEXwthEAJuWRRhchF+lL7nQ5PfEq5QGCCH3SeNh6UmYRSBwJPBwxuBKXdc98yqO5uVIOP3iw6hP/UFlvEhLr2Cl7q7lKf89F3njlC6ptVR5ljZ7C6rF6kmwjsoLsalEKuv/vwfOkKAqILryAN+4JPxK3ZEiO6G2kAOFp2O3uMEY/rQ/wGvD8NyOvnu0CXtSoC6cf2xwH9nbrB6nQuDGYfNfuRhF7IUqZi6uz7G6YVXbEClfJC6TPKNXDGxoPUaVowa1ul+Ruz56SfN8t6dlqM/aVz4u/Tp68ciStTZmte68LOWC06xjDTpaR+JxFxtG/YalpitbKids65JS00JrWpabsQrs8y35Qyjr2aXmKQJUA9s12n/dPzpmXFPNxOoKY/S/DxnqjQ49eG0P3iFsJxd4jHP03oCvn6s2ox X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MN2PR10MB4112.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: p08QpdzhpNpmEUxVNOxLO50NmAcsK1/PaeWrDkxFo++4GcdkCbBQGFQlLWBrbm1ehFPyUtBEveg3b+KzhY5aEfQNWXiR2ekc1C3lXeDpoZk5L6K0Quvxx50TTIemdy6QhoAzWRHfFqchqCWdIWtxcSeh+XFXIG9YIHMDAYYkOSncqe5tzmEVaEowl2NWyxH7vX+QtndZuc42JYQ6gDd2Vh6VmufDpqzM7FdG/PFHWQoctYd3Qx38lW1O9+3kQsQqcDVnHDW/xQjsx1R0xgi9U3EhQmLovHkA5SDLl8Xczaki/NHNJa/GZGF/u6+WhOd7IMpPZ8Je4TBasl8QvPtqw3WzcZHma0+5fXMyoCkLAmSAxmpMFcj27eRypmJ7eSyJ5c0QxT8HFfY96BD/bPjNsw/7ou6qdvli0it4554QIaiQYaQZ31MIiyWuEeN1LpXZTX1/4Fm+r5Cg5osB8YdsgK+zk9tPmEmd1MfhpAbL47bNU23WX1oKo1vbyqSKV3Sw9ifUNP5113V9ppbljm4RVQIoz4aaWUdzwvRbCbzS7qo0oNTh4A5w6U8g88UECvoR3FpJ+ePsaN9tv/FR0f5MWq0wVlONh6qM22gRf+83oKGnA92qpoTywHYRvKoshODvVt5/kB/a/XueTxuewXfMt7pm0r9XD8qDy6arI5hsm9KztlAXzbNeShS0dBkDmZFEdhTU2NLp93NKXWIjiHDL2bk4VYqBPJBNNv/dj93Elhsov4K3Ks0C4RbrrB2HWDym+qs5fL1twd0DqHyzIheW9CJ6ZfGfPe4jmz6jyvNHAkbns+BDahy8QMoyDMzv5+ljjFf3GNMaTfzwpTxashPmndRw2Xte220tlIUFmDcg71AVuo37yWYNVVVj5XqKTT/cypbokRAotpjMcIhXDEc7n7BkSG10sWs90nQSKIWIZlJtOMcGr394y5LysXfDaDRUKtOIHHiufhF6EfgtinXcoktH46q9fWUjdVj7Q8meuOhv/KU+CC+pkpj93svQKh5NHsmHfN07fAC6TbPidwovF4cC4AQxIaxcnJGphP5kh4H877NN2mDkI50YxY6yHXnXBD+H6LkhRhVEaTbj4rzS8h8EUwQfs41A016gMD56zzYAKWDp+lmS+Mgv+LrfSDS9OTtSabyiFmvdHRsp/8k9FQcdkQLlo6mmTD/NA2NkmoBmYF/9eXuDhwqT03EMkvXr5kmmgWxYsA0tm5GgcocPsSwbfFtCML6E275Rp23btk1jKRRflFVAf12X3j44cRIDZjin0E14MY1fVnOlcQiM+NbF30BHdc34VYijC9Z0mK181/j6tf0Kwhh7pO9cUk2Zi1b/2H8ayxYt+Crz96h0N6/OofNTDJQkL7qOX2VM0qlWg50dDMZ2JN2xcFEGYf5NeWaL3lPNeboHXvO2pcufZSCbZW3YR2M06sRJILtOoE3gtbbL+de5KvYt66NrArvOgM4VZjpKpYiAEs0G17RB93HyQXdQ5m2iX5TcY/vexFlHErfaQEa0N16b9T7hyGVg0654erYs6Zl/q72/Dxb32tFY2iX+fWzlxgK6vQe2H2Jwcou1keSES/FOurpgEn91XDP1vqcC9oB8VLmCUWC0sw== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: ldHCv+oePHu5gZsCazBSBZfQtuFL432SSXnjoX2AMZe3vqKBABkkVbsKDIYJ9sbDsb/D68eg9YyQRAcUh0vxVIuCMW8/56mwM2wzLvWXSuriEiQT1mN5AfOnPSCvLVGFYhzK+AY95291RCYqyLT7vTTp+9RVz70kkDNf6Byx8XG1ZV+xJ7gFqM72ha3OfCdx57hO2zROQaoCrXcYh/9OE2ZLhGem6lPZ/B7HJximL87R3WQHojtFVWe4drze/geAM0tMBRCKb9itBdvufWr+aW9HjjI9r8G97byUyQrpGzvgRTFlTIELR+aha5NWmcdz32By9liip3STQq/meNujSDthpYWa5nObEBVaCx5AAhsHp11+UxRaVkaxgyv/SJk7LLTKQXOwexmqFO11E4mHGwQyT3BsA3mJ3iwjWwbnpyJL0d3h2odQ5zGuysjbCSrWvWdNAULZ+Gjs7HimV76EpvuKZJgyoDaRvxSjsnBbE9GIhyuGIcuKNvRhzmwjIAUhehS3+n0IE/CgGUf4uMvl7MABIZfHzeidr5JgR7+PBVsOVWQ68PqcpTC5o03CwGWhilK2WMJiPpTOni30Ulc5ev1jkVDucnmU+5lQCLN+6n8= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7e834499-ff8f-4cfd-69ef-08dd5c9a6f43 X-MS-Exchange-CrossTenant-AuthSource: MN2PR10MB4112.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Mar 2025 10:34:11.9864 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: kaZrh3DYFpzW/kOgs9xM9gBS1AjPR0Yo/T0GSNI1a8CydT9wB3JlNutrvq8a7edsvoVMSyNdMTSqxlcZCX375xv/GwSHOnKB/hatJe945ug= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB4647 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1093,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-03-06_05,2025-03-06_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 adultscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502100000 definitions=main-2503060079 X-Proofpoint-GUID: ff0Vst4zLjB1E6usFDdq-BB48XmosC4U X-Proofpoint-ORIG-GUID: ff0Vst4zLjB1E6usFDdq-BB48XmosC4U X-Rspam-User: X-Stat-Signature: 94i95h16oyfi7buswudofoobeoi7xy96 X-Rspamd-Queue-Id: E498A140007 X-Rspamd-Server: rspam07 X-HE-Tag: 1741257256-817712 X-HE-Meta: U2FsdGVkX18zCJkDB3Bnb+fc10QxqxYMpq3DlY0sS3AejZG9eubfGPPshmJo72Oe/fCuU5jgZPR0JtsmzVf3bxlICL6VdgZglyeeKwPSof/o3Y9a7I7SmL0yXVOp5dFc/aU142ATCJNJvUkMAIdvwovdJuiLXGiOeX9R3dad5e9CLWNcrqNiFyklgKd+bBbP2du2OS7ynJyKmLCWSmBMvC9Ws8kowsipd6BjfLM4oe4/LBAz8Uzmic649PMUKU5FSiFDaddThkRcBOzrE/Ylx32O0WzmeEluM9ksR/WJlOrXwY+jxBQ7LgBy3guMm5sHAm5oMgTK5jU9fSy8N1VjuA7R+TVgheHTtnYK+sopskPowtnH86+628FWokVYLeVn/dsTm+79fGfG575ZG1WLmoFGYVD0Mzad3E4wVmsipiHYjbITRacE83DYqN1oilQ+lHnnq04DJno3ZkMRO8NEW16H/58+Bbt+1EYArem9ACB43Kh82386JWhngHY0v5z/QIdhCGWPrpKo9gWuFpFjI5LGpZNCKDmABcCce/CseK5dM1ybAgar/mIZYsU+NrGuvQzbdmPXiKydBNCUjTsboloScpNL3L8yq9r2WGcjcMna997AvtflTF47f/t7oZ5mi7Nya6V3JNbuNSDcr8WoIG8RLpqx8qkmSzt+raMOaRKcG/j7gLr/x69zNPgxCR+MHljx7cGJ7kImkmF6c/LkrfdOefmxEkbiLR7uvwk2NWEBIpLGM2O0IbevR49skco9CKxjLlRZUQNhZf46DLQFP383tQQE/de2S3zJ4N7j1J8j/sloH6dkiaBJgj1g0D/T1KPs8VpKzeCsg0yeS6zHe0ZGl2+vgnToIQ5yyt7wYxU/nO04V8zwSjgB5yxXhp8q++C9JllS0s08Jijx6ml8xJ7/6loMftgFEJRLpbeVAT9eWLZO07KEujyeRauWnGoSCWRg4Jj6NN1FJiuCqYp LaOAfeS8 Evzo4KRTB2b4N8aDk5oBpSQ3DEKl8fmdCr2Mv1jx515NbtYTP6oWAZ5CrjciP1uCRxYcec7G0ZdLXctEid/refQG929corlqmQB0HGy8PxadMc12n1KbV/qP25NrhRTAW98QiRmcQhEOPRmQGkP/md+ITJw8TguNIwDObgCkp23V4CDmcavywK2IVw2kj/NpWryDgBlYNA5nkSl4Bc7KbMAudlWlmLquXtVIZHJJVq4PwKyeowf4w2SXcxc9W6PVoQv923kYx+BpEIm3bNxVq8JAqkiKNU/l43WgdYQU4m5UtzTV14KbiOK+KJIXSIbK4m9z+ZLlsqVcq3nFqBWwVnWWJLQuTQb3quDxI+wt+qzCVgxImp7aJ7qV6yB+2npCnh6QzYedeUEpx++HVNo0HOsTThq7PXlZ+MlhTx97wRTrO5hM40z0Z3dGvHlwN25Ju/ZJwzGZ37IGhELnJ+d6tKkFP7pS9ampBSF3LxxmvgGh+o4YIzNIzbMHX9a559N+MqLGuAYVI5/qycocRra8ShonSNXkcbUY1Y5FjUW3ewA51NNbzkhBcNBBAu6+iGUEXH4+UmZNfXknDZ5M3lQug+anlnVbXr7TrNCdqxv9sJMUWvjw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Place checks into a separate function so the mremap() system call is less egregiously long, remove unnecessary mremap_to() offset_in_page() check and just check that earlier so we keep all such basic checks together. Separate out the VMA in-place expansion, hugetlb and expand/move logic into separate, readable functions. De-duplicate code where possible, add comments and ensure that all error handling explicitly specifies the error at the point of it occurring rather than setting a prefixed error value and implicitly setting (which is bug prone). This lays the groundwork for subsequent patches further simplifying and extending the mremap() implementation. Signed-off-by: Lorenzo Stoakes Reviewed-by: Harry Yoo Reviewed-by: Vlastimil Babka --- mm/mremap.c | 405 ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 251 insertions(+), 154 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index c3e4c86d0b8d..c4abda8dfc57 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -942,33 +942,14 @@ static unsigned long mremap_to(unsigned long addr, unsigned long old_len, unsigned long ret; unsigned long map_flags = 0; - if (offset_in_page(new_addr)) - return -EINVAL; - + /* Is the new length or address silly? */ if (new_len > TASK_SIZE || new_addr > TASK_SIZE - new_len) return -EINVAL; - /* Ensure the old/new locations do not overlap */ + /* Ensure the old/new locations do not overlap. */ if (addr + old_len > new_addr && new_addr + new_len > addr) return -EINVAL; - /* - * move_vma() need us to stay 4 maps below the threshold, otherwise - * it will bail out at the very beginning. - * That is a problem if we have already unmaped the regions here - * (new_addr, and old_addr), because userspace will not know the - * state of the vma's after it gets -ENOMEM. - * So, to avoid such scenario we can pre-compute if the whole - * operation has high chances to success map-wise. - * Worst-scenario case is when both vma's (new_addr and old_addr) get - * split in 3 before unmapping it. - * That means 2 more maps (1 for each) to the ones we already hold. - * Check whether current map count plus 2 still leads us to 4 maps below - * the threshold, otherwise return -ENOMEM here to be more safe. - */ - if ((mm->map_count + 2) >= sysctl_max_map_count - 3) - return -ENOMEM; - if (flags & MREMAP_FIXED) { /* * In mremap_to(). @@ -1035,6 +1016,218 @@ static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) return 1; } +/* Do the mremap() flags require that the new_addr parameter be specified? */ +static bool implies_new_addr(unsigned long flags) +{ + return flags & (MREMAP_FIXED | MREMAP_DONTUNMAP); +} + +/* + * Are the parameters passed to mremap() valid? If so return 0, otherwise return + * error. + */ +static unsigned long check_mremap_params(unsigned long addr, + unsigned long flags, + unsigned long old_len, + unsigned long new_len, + unsigned long new_addr) +{ + /* Ensure no unexpected flag values. */ + if (flags & ~(MREMAP_FIXED | MREMAP_MAYMOVE | MREMAP_DONTUNMAP)) + return -EINVAL; + + /* Start address must be page-aligned. */ + if (offset_in_page(addr)) + return -EINVAL; + + /* + * We allow a zero old-len as a special case + * for DOS-emu "duplicate shm area" thing. But + * a zero new-len is nonsensical. + */ + if (!PAGE_ALIGN(new_len)) + return -EINVAL; + + /* Remainder of checks are for cases with specific new_addr. */ + if (!implies_new_addr(flags)) + return 0; + + /* The new address must be page-aligned. */ + if (offset_in_page(new_addr)) + return -EINVAL; + + /* A fixed address implies a move. */ + if (!(flags & MREMAP_MAYMOVE)) + return -EINVAL; + + /* MREMAP_DONTUNMAP does not allow resizing in the process. */ + if (flags & MREMAP_DONTUNMAP && old_len != new_len) + return -EINVAL; + + /* + * move_vma() need us to stay 4 maps below the threshold, otherwise + * it will bail out at the very beginning. + * That is a problem if we have already unmaped the regions here + * (new_addr, and old_addr), because userspace will not know the + * state of the vma's after it gets -ENOMEM. + * So, to avoid such scenario we can pre-compute if the whole + * operation has high chances to success map-wise. + * Worst-scenario case is when both vma's (new_addr and old_addr) get + * split in 3 before unmapping it. + * That means 2 more maps (1 for each) to the ones we already hold. + * Check whether current map count plus 2 still leads us to 4 maps below + * the threshold, otherwise return -ENOMEM here to be more safe. + */ + if ((current->mm->map_count + 2) >= sysctl_max_map_count - 3) + return -ENOMEM; + + return 0; +} + +/* + * We know we can expand the VMA in-place by delta pages, so do so. + * + * If we discover the VMA is locked, update mm_struct statistics accordingly and + * indicate so to the caller. + */ +static unsigned long expand_vma_inplace(struct vm_area_struct *vma, + unsigned long delta, bool *locked) +{ + struct mm_struct *mm = current->mm; + long pages = delta >> PAGE_SHIFT; + VMA_ITERATOR(vmi, mm, vma->vm_end); + long charged = 0; + + if (vma->vm_flags & VM_ACCOUNT) { + if (security_vm_enough_memory_mm(mm, pages)) + return -ENOMEM; + + charged = pages; + } + + /* + * Function vma_merge_extend() is called on the + * extension we are adding to the already existing vma, + * vma_merge_extend() will merge this extension with the + * already existing vma (expand operation itself) and + * possibly also with the next vma if it becomes + * adjacent to the expanded vma and otherwise + * compatible. + */ + vma = vma_merge_extend(&vmi, vma, delta); + if (!vma) { + vm_unacct_memory(charged); + return -ENOMEM; + } + + vm_stat_account(mm, vma->vm_flags, pages); + if (vma->vm_flags & VM_LOCKED) { + mm->locked_vm += pages; + *locked = true; + } + + return 0; +} + +static bool align_hugetlb(struct vm_area_struct *vma, + unsigned long addr, + unsigned long new_addr, + unsigned long *old_len_ptr, + unsigned long *new_len_ptr, + unsigned long *delta_ptr) +{ + unsigned long old_len = *old_len_ptr; + unsigned long new_len = *new_len_ptr; + struct hstate *h __maybe_unused = hstate_vma(vma); + + old_len = ALIGN(old_len, huge_page_size(h)); + new_len = ALIGN(new_len, huge_page_size(h)); + + /* addrs must be huge page aligned */ + if (addr & ~huge_page_mask(h)) + return false; + if (new_addr & ~huge_page_mask(h)) + return false; + + /* + * Don't allow remap expansion, because the underlying hugetlb + * reservation is not yet capable to handle split reservation. + */ + if (new_len > old_len) + return false; + + *old_len_ptr = old_len; + *new_len_ptr = new_len; + *delta_ptr = abs_diff(old_len, new_len); + return true; +} + +/* + * We are mremap()'ing without specifying a fixed address to move to, but are + * requesting that the VMA's size be increased. + * + * Try to do so in-place, if this fails, then move the VMA to a new location to + * action the change. + */ +static unsigned long expand_vma(struct vm_area_struct *vma, + unsigned long addr, unsigned long old_len, + unsigned long new_len, unsigned long flags, + bool *locked_ptr, unsigned long *new_addr_ptr, + struct vm_userfaultfd_ctx *uf_ptr, + struct list_head *uf_unmap_ptr) +{ + unsigned long err; + unsigned long map_flags; + unsigned long new_addr; /* We ignore any user-supplied one. */ + pgoff_t pgoff; + + err = resize_is_valid(vma, addr, old_len, new_len, flags); + if (err) + return err; + + /* + * [addr, old_len) spans precisely to the end of the VMA, so try to + * expand it in-place. + */ + if (old_len == vma->vm_end - addr && + vma_expandable(vma, new_len - old_len)) { + err = expand_vma_inplace(vma, new_len - old_len, locked_ptr); + if (IS_ERR_VALUE(err)) + return err; + + /* + * We want to populate the newly expanded portion of the VMA to + * satisfy the expectation that mlock()'ing a VMA maintains all + * of its pages in memory. + */ + if (*locked_ptr) + *new_addr_ptr = addr; + + /* OK we're done! */ + return addr; + } + + /* + * We weren't able to just expand or shrink the area, + * we need to create a new one and move it. + */ + + /* We're not allowed to move the VMA, so error out. */ + if (!(flags & MREMAP_MAYMOVE)) + return -ENOMEM; + + /* Find a new location to move the VMA to. */ + map_flags = (vma->vm_flags & VM_MAYSHARE) ? MAP_SHARED : 0; + pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >> PAGE_SHIFT); + new_addr = get_unmapped_area(vma->vm_file, 0, new_len, pgoff, map_flags); + if (IS_ERR_VALUE(new_addr)) + return new_addr; + *new_addr_ptr = new_addr; + + return move_vma(vma, addr, old_len, new_len, new_addr, + locked_ptr, flags, uf_ptr, uf_unmap_ptr); +} + /* * Expand (or shrink) an existing mapping, potentially moving it at the * same time (controlled by the MREMAP_MAYMOVE flag and available VM space) @@ -1048,7 +1241,8 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, { struct mm_struct *mm = current->mm; struct vm_area_struct *vma; - unsigned long ret = -EINVAL; + unsigned long ret; + unsigned long delta; bool locked = false; struct vm_userfaultfd_ctx uf = NULL_VM_UFFD_CTX; LIST_HEAD(uf_unmap_early); @@ -1067,70 +1261,38 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, */ addr = untagged_addr(addr); - if (flags & ~(MREMAP_FIXED | MREMAP_MAYMOVE | MREMAP_DONTUNMAP)) - return ret; - - if (flags & MREMAP_FIXED && !(flags & MREMAP_MAYMOVE)) - return ret; - - /* - * MREMAP_DONTUNMAP is always a move and it does not allow resizing - * in the process. - */ - if (flags & MREMAP_DONTUNMAP && - (!(flags & MREMAP_MAYMOVE) || old_len != new_len)) - return ret; - - - if (offset_in_page(addr)) + ret = check_mremap_params(addr, flags, old_len, new_len, new_addr); + if (ret) return ret; old_len = PAGE_ALIGN(old_len); new_len = PAGE_ALIGN(new_len); + delta = abs_diff(old_len, new_len); - /* - * We allow a zero old-len as a special case - * for DOS-emu "duplicate shm area" thing. But - * a zero new-len is nonsensical. - */ - if (!new_len) - return ret; - - if (mmap_write_lock_killable(current->mm)) + if (mmap_write_lock_killable(mm)) return -EINTR; + vma = vma_lookup(mm, addr); if (!vma) { ret = -EFAULT; goto out; } - /* Don't allow remapping vmas when they have already been sealed */ + /* If mseal()'d, mremap() is prohibited. */ if (!can_modify_vma(vma)) { ret = -EPERM; goto out; } - if (is_vm_hugetlb_page(vma)) { - struct hstate *h __maybe_unused = hstate_vma(vma); - - old_len = ALIGN(old_len, huge_page_size(h)); - new_len = ALIGN(new_len, huge_page_size(h)); - - /* addrs must be huge page aligned */ - if (addr & ~huge_page_mask(h)) - goto out; - if (new_addr & ~huge_page_mask(h)) - goto out; - - /* - * Don't allow remap expansion, because the underlying hugetlb - * reservation is not yet capable to handle split reservation. - */ - if (new_len > old_len) - goto out; + /* Align to hugetlb page size, if required. */ + if (is_vm_hugetlb_page(vma) && + !align_hugetlb(vma, addr, new_addr, &old_len, &new_len, &delta)) { + ret = -EINVAL; + goto out; } - if (flags & (MREMAP_FIXED | MREMAP_DONTUNMAP)) { + /* Are we RELOCATING the VMA to a SPECIFIC address? */ + if (implies_new_addr(flags)) { ret = mremap_to(addr, old_len, new_addr, new_len, &locked, flags, &uf, &uf_unmap_early, &uf_unmap); @@ -1138,109 +1300,44 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, } /* - * Always allow a shrinking remap: that just unmaps - * the unnecessary pages.. - * do_vmi_munmap does all the needed commit accounting, and - * unlocks the mmap_lock if so directed. + * From here on in we are only RESIZING the VMA, attempting to do so + * in-place, moving the VMA if we cannot. */ - if (old_len >= new_len) { - VMA_ITERATOR(vmi, mm, addr + new_len); - if (old_len == new_len) { - ret = addr; - goto out; - } + /* NO-OP CASE - resizing to the same size. */ + if (new_len == old_len) { + ret = addr; + goto out; + } + + /* SHRINK CASE. Can always be done in-place. */ + if (new_len < old_len) { + VMA_ITERATOR(vmi, mm, addr + new_len); - ret = do_vmi_munmap(&vmi, mm, addr + new_len, old_len - new_len, + /* + * Simply unmap the shrunken portion of the VMA. This does all + * the needed commit accounting, unlocking the mmap lock. + */ + ret = do_vmi_munmap(&vmi, mm, addr + new_len, delta, &uf_unmap, true); if (ret) goto out; + /* We succeeded, mmap lock released for us. */ ret = addr; goto out_unlocked; } - /* - * Ok, we need to grow.. - */ - ret = resize_is_valid(vma, addr, old_len, new_len, flags); - if (ret) - goto out; - - /* old_len exactly to the end of the area.. - */ - if (old_len == vma->vm_end - addr) { - unsigned long delta = new_len - old_len; - - /* can we just expand the current mapping? */ - if (vma_expandable(vma, delta)) { - long pages = delta >> PAGE_SHIFT; - VMA_ITERATOR(vmi, mm, vma->vm_end); - long charged = 0; - - if (vma->vm_flags & VM_ACCOUNT) { - if (security_vm_enough_memory_mm(mm, pages)) { - ret = -ENOMEM; - goto out; - } - charged = pages; - } - - /* - * Function vma_merge_extend() is called on the - * extension we are adding to the already existing vma, - * vma_merge_extend() will merge this extension with the - * already existing vma (expand operation itself) and - * possibly also with the next vma if it becomes - * adjacent to the expanded vma and otherwise - * compatible. - */ - vma = vma_merge_extend(&vmi, vma, delta); - if (!vma) { - vm_unacct_memory(charged); - ret = -ENOMEM; - goto out; - } + /* EXPAND case. We try to do in-place, if we can't, then we move it. */ + ret = expand_vma(vma, addr, old_len, new_len, flags, &locked, &new_addr, + &uf, &uf_unmap); - vm_stat_account(mm, vma->vm_flags, pages); - if (vma->vm_flags & VM_LOCKED) { - mm->locked_vm += pages; - locked = true; - new_addr = addr; - } - ret = addr; - goto out; - } - } - - /* - * We weren't able to just expand or shrink the area, - * we need to create a new one and move it.. - */ - ret = -ENOMEM; - if (flags & MREMAP_MAYMOVE) { - unsigned long map_flags = 0; - if (vma->vm_flags & VM_MAYSHARE) - map_flags |= MAP_SHARED; - - new_addr = get_unmapped_area(vma->vm_file, 0, new_len, - vma->vm_pgoff + - ((addr - vma->vm_start) >> PAGE_SHIFT), - map_flags); - if (IS_ERR_VALUE(new_addr)) { - ret = new_addr; - goto out; - } - - ret = move_vma(vma, addr, old_len, new_len, new_addr, - &locked, flags, &uf, &uf_unmap); - } out: if (offset_in_page(ret)) locked = false; - mmap_write_unlock(current->mm); + mmap_write_unlock(mm); if (locked && new_len > old_len) - mm_populate(new_addr + old_len, new_len - old_len); + mm_populate(new_addr + old_len, delta); out_unlocked: userfaultfd_unmap_complete(mm, &uf_unmap_early); mremap_userfaultfd_complete(&uf, addr, ret, old_len); From patchwork Thu Mar 6 10:33:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Stoakes X-Patchwork-Id: 14004150 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A781C28B22 for ; Thu, 6 Mar 2025 10:34:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 67EE2280005; Thu, 6 Mar 2025 05:34:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 63078280003; Thu, 6 Mar 2025 05:34:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 410D4280005; Thu, 6 Mar 2025 05:34:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1C5A7280003 for ; Thu, 6 Mar 2025 05:34:22 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id AF0001CB80C for ; Thu, 6 Mar 2025 10:34:22 +0000 (UTC) X-FDA: 83190766764.24.1A4E338 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf16.hostedemail.com (Postfix) with ESMTP id 5E166180004 for ; Thu, 6 Mar 2025 10:34:19 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=KVGah5E6; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=gujjvwQi; dmarc=pass (policy=reject) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf16.hostedemail.com: domain of lorenzo.stoakes@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=lorenzo.stoakes@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1741257259; a=rsa-sha256; cv=pass; b=ZK7y9AGfCcCdLseoCgw7SXmw+qEy1ExmGePyfPjcnmEndh1jbMhS6bEtzYlnLFuiZF/NDf hrPKJiis1a/Hqs+OH16aLYoHium3qGPA26q9K9NvjUAU8dIp4yl4Kq8dC/59H8B8z0lBOK xJDZOScvGP6Z784pJv1k5UFJK7vIsBE= ARC-Authentication-Results: i=2; imf16.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=KVGah5E6; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=gujjvwQi; dmarc=pass (policy=reject) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf16.hostedemail.com: domain of lorenzo.stoakes@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=lorenzo.stoakes@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741257259; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5OOk1sOhqeNPU01ZgtOnGB0jLypwuBEMjd32gqcDYoo=; b=JgWPG3hzWTBUUNZztwrQYroEIRnbZtJaVe1EiUV3/B+gh06yzbvv2wcjahUStgcVdSFVEZ g8Qj3GXY535YMOze4Rg/nqkMsUaCfQVZQPCEVPn5oXSTEOTZ59MaAI747s49lwjRKDRMhj 7HL7t4qwTQpSs9CM1PZt2ehe5QYNYKA= Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 526ABfhN024547; Thu, 6 Mar 2025 10:34:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=5OOk1sOhqeNPU01ZgtOnGB0jLypwuBEMjd32gqcDYoo=; b= KVGah5E65sHnvkWoNY2hAwWLbfQixlQbjXmQRryBGWy2eyKEdRuQl3dkwB0jGVN2 SvsJcTAUwuHvmBb9Pp8bnKKmttRwGjTzAThQ9vBts0NktcqpvxdYMLebzKWtkXmW txJpspEMge2voMFi2N1o8BHFQDrQDpUTqnuG1PnhWMgkZQAPchXr2e3VbtebZvbL nfIeBU9wtD0AKIn8P5wZ2UF12VrcxOBYBZZNFMECKO84ntAg1r0nliMSSEJTucJD 4mysf82rI3nlJCp4KdmMoC/sff8gykqU1ajSyMlTVJBR9b2XSj6XlLLjfb0Y6aB+ FaiIgGzznYNohHVlDUpQKQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 453u8hhttd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 06 Mar 2025 10:34:17 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 52693cEn038319; Thu, 6 Mar 2025 10:34:16 GMT Received: from nam04-bn8-obe.outbound.protection.outlook.com (mail-bn8nam04lp2044.outbound.protection.outlook.com [104.47.74.44]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 453rpj67vg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 06 Mar 2025 10:34:16 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=GakV068XVdrNs7qWV48BLUrK9/9FHo3It7MhATPDZboOrJ1nptj+7BqQul0NU44re9cFacyF84/LV04QFuZ6pjo6pqd9smNb+NR9WnG9PO/5A9bgLhUpz4qBtIdF6REmrGg/CqsTzqsSpw3vPCPBllVChPUwWwPx8uu947igzo0f36SY2tNblxHgZ7Qll8g14BeY9JRzt8qbHRrDkDMeg3w6lgzTueRx3vhonMEruYSAAn79YdGFUrUzS0c5qGN6s1gfLuSeOT5GRHdc+D1I7rnEC+ghjXMN4/cKXvXWRBIyuN23IPbBTBbmYX9Z3m4wA30m3mNM6QGLYT2eK1US3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5OOk1sOhqeNPU01ZgtOnGB0jLypwuBEMjd32gqcDYoo=; b=BZu4OPitaTN/1C16d8wJGi22UnfPXPBFRO8VrE867ZerC5upBrvfoKpn/nHKPQYmZGNL8UHnpxjvHPL5HsodD0aN9/uyzVwOiLGdwQ588EnrBAFu9pD5R08gNRH3oRQ7OCGdT7zHe4PRDd/FMP+m/e9Fsa4i2fJC1VBg2W6VaWtx2xu3peiluuzowauRLyASOhwrEYTIRjAYqB372Gvuqc5lJV4pbjeSyq6iPWAbUXsDMmPKj0knahOcZ1RvGqnRL1kHBU8B2Q8IUnp5ciZoznakmlwoJcWypwFdslgdYXGZ76SknPspirm5mdcHeSlazEwhz8XiU9hk+UL6H0+z7g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5OOk1sOhqeNPU01ZgtOnGB0jLypwuBEMjd32gqcDYoo=; b=gujjvwQibPbvRTP9KH6h1QY9XnEWvTWtdtw+a7RB20mruyGG2WmPCvXlDs5wYm4fd0AzkQlQxx9JZ9enn/9+yjqZ8gnFhztH5rX3Lgs/hzuBmiieLGzV2OPEDdLFj7qMQgBCrEGHXWvacqzYV8kZtTRG1ZlITBvMZIyZNPGVTU4= Received: from MN2PR10MB4112.namprd10.prod.outlook.com (2603:10b6:208:11e::33) by PH0PR10MB4647.namprd10.prod.outlook.com (2603:10b6:510:43::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.20; Thu, 6 Mar 2025 10:34:14 +0000 Received: from MN2PR10MB4112.namprd10.prod.outlook.com ([fe80::3256:3c8c:73a9:5b9c]) by MN2PR10MB4112.namprd10.prod.outlook.com ([fe80::3256:3c8c:73a9:5b9c%7]) with mapi id 15.20.8489.025; Thu, 6 Mar 2025 10:34:14 +0000 From: Lorenzo Stoakes To: Andrew Morton Cc: "Liam R . Howlett" , Vlastimil Babka , Jann Horn , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Harry Yoo , Yosry Ahmed Subject: [PATCH v2 3/7] mm/mremap: introduce and use vma_remap_struct threaded state Date: Thu, 6 Mar 2025 10:33:59 +0000 Message-ID: <1aeadc40d377fff8796b7c114cb0351c92a68f20.1741256580.git.lorenzo.stoakes@oracle.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: X-ClientProxiedBy: LO2P265CA0372.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:a3::24) To MN2PR10MB4112.namprd10.prod.outlook.com (2603:10b6:208:11e::33) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN2PR10MB4112:EE_|PH0PR10MB4647:EE_ X-MS-Office365-Filtering-Correlation-Id: ff49a02d-8832-489c-b199-08dd5c9a7080 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|366016; X-Microsoft-Antispam-Message-Info: LiMKxbqt6XoP1bqhmBe27bdhUg5W7KLo1hC6FuN89uSOX+M6HRswZi9fX0UeHQAbYv/29Mm2E28DF3TXyQU5K2uj+2QfMftBGzY3YZ4wNhl1o0n1v04JaonGKLefmf546HIsmfcCSrYJsU2YFurrFK48dtJog2jtedEGGzF3YbLMBl95vLdgdPnbtGkSag73pPrIgREBt1MpkDK9es2fIiFv9ufWY8y+eARbho+oV5ZweV2lMeMnW94jU0THfCf0ZztlpGKBMQuV/GOEYLWPHL2/puqerFDtZj6qrNSdY04VpzW2H1T1NntOhym5Vhn9paPzIj6gID6Md2yfTm6RL9CG76zNhWyWe8oCchF63flatJzf0rZeiJTfHUpQZIU/oxhYP73DmVTerp0h4oxybFsp0t8DtR1+sLwsw/b2D4Dv+ZreJ6WPRD0nwRfLLULQhsLyB4U1RVgKi/etb88+o76cjYb97KP+5qkU20UxWfHdR47KogkJGBTO4oBOI9/NUXKWNNCT2DcPjdIBtxAdBxviCdvJ5ztq4kxy5mmWDtViAowR0tIsc1GpiT5udrdAS6UgoOYjPF4EUSwdZcsE4ovUyOr+hsYMZFracYAuoDkhpa44FDm7/AvEkX9IdzNb+plvBkREMKkEx5lHUCN2aJAvF74LCX6+ZEArXCBxI0WpyzITVpryQDj/iX8W1/CzcadEr0bVv1ZBFb1kx1rQhotfyz1bS6qTPomnfLJ8FjTw0qZp1rB77T/ypeQ3ejynI+l6MOX13C2Bf1RLjidPpVKYzO4KwWXoE86abQ/N+lURVbFGTHkZZXsrI5vVna4FElRDpbGqZAAwqkrZjYDqBe1uBGN8RRijdiukWo83LjdbZrEIoPxDhHbROq6Q0DXqWPNOrhgeXunxoxVIq3ULntvL3iemK+9xWvibfdg2UNzqOWuTCHXczwL1PrKB8V0lHJc4xoSmhbBLJOQaADIAHDM70TO1V73GyWq8KtAskQj27QVCP20rZGCnm9NLp5Zt2u/D/FUmVOJsaGEdpcH9/KnMxdoUGHZz+i3RdFV0yFb8re8Lw1ubcP3dc1eE88ypDnwcYwD4OaykyKQ++2gdRiWDcxDcGEXpKvObhCRwxRAPkSfoI5nuL+guvkmddf458vQ5GDBlGso8L8JbouQEK/EiekdLPJjV8UiaW1cHtRGHeQnaBBw7/iMjI68n3pagEPhnnWS0Y6C6uHdzKFQCSZ+CsDW7JRQsCK6z4oMWvSUsPCKAley4Hp40ZK3uTCL07tfU3rnzL4gWCyEhzueUBRTa41yLAmqtq8lCHlllznHD/WQo/6wTTXF0NwsT6M/W9cjk/9aUYYF1sZS+P9lQliIy8WN0KHepdiCL7+9DIdNeszEs/QX9wo7Q7zx28ipj X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MN2PR10MB4112.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: sE4bkTSiGU4m72685KkObvOJwOSQ6CYurAUooJ/Ak7bgQsd2aA6J7NqStBK53/St/Lf7ArKoOkOAybvqHgZ4Yi8sLEeHa2kDACK+ZBnJElzFezL6WpqUxEZ0UDqGIPn8IWQ7ZMp7mVZ8JWOsxiWy6Fmpjor1Wcc8dZdmKnUFyTNaAOxb3xEciM2RihJ6jr0PZYe5QvMfsNwQEfDprD9tarN2i29dCtpb9ZP3Fl/VE+UlSfe/+QClbUUzuOWD9+Rgab5BMAKCvAAa6i0A6y7UH+kR07UDAB/3LqT7wsm1lZpOu6EBSy8kePjErCIBsg9rG75NaUdIUjmfBwB98PSWDvMsb2tsEFzTFFOx0NNzLT37MxmYUaGlAoy1+DOseiFN0V6c2ZWCT50arwIqPua86zYuIBeyywKPNVaRcYegiS+B2hVzC5Umtmp+QEiELCEVEkGxVSFO4HWwCW5NXl2e7p0Fw7AqMReCfpRI8kBoDRsAUoHXsmTy0fZW6NBLVW3dOzDBi/0J2oM2Np1VzGS8zQKiJRInGik9wbGioaMhiHA9HXbTeDcygZYvaeBNQH4hLEgSeJ2NFevlcv3TVMp4ikwvc74DmynFTz/c/X6NEV8VSAmADUgiHB4B4bn8+/B6yFUgb3jMCdfH/SNZsDMWxOi6QBXbRe9QwR6QScSljB3GMt3XASuywjZupQ0DUS2CmWwcJWjYe24WvCMaYwTJ51ImC9ReznXeTMeGU2eF3TzdsTSlhUnCJwYA4DfwYCC6fM0vFJ/Y4oRUh0kJtVCHat2VZ8DMKIt1c9WUYCTK97jPqo/sjbFvPHR+yTRgs9hamjNHAEgTjlhLYxLjN38y66flTOH7pIVr8qDAIPoiCk/RKvVXzCf2n4+QtMr1QCduvba1usC3ZzPgOOAuRCZLqwa9ELjBWtvxp5p2b6gia8GjrL/X8w9rGSE8zCocV1bj5uNhvv6V4HC9urXdRed27S/rZvRUBINnQpSx/qLIdW3vWpbAmoBz/nMf40EnuiKlnIeCP+S38juhuXHm0xa/dtVmGTlY6lgr6lp0hx2H8JDYMsa1onipOWfyZUyfnHY4BEdK0Ey98tPaHgSCVWafXv+i1EJr6mw6D+oSzDQfIuOHe01muvbgCuxrTMmTBrHuXlJcSig0A9rHO8iVPRLo3OhJTyr+v0GqOBPqaK1lte4wqG7zYd4B4oiltVV1QNRjV6qbSKSP3CfAN3I6X3wGvvfPenIwJpBqDH/AjlVCzp9MZ7JUv9he/PtVcSNGF4fNpz3m7bjprhkkAlyz16ylBquxSneZC2kPXqtiKb+LKa4PrcqLdd/b9D5A7pJhdhBxUMi5pdciBnMQfKRDkyfqfitGsYpLAWDGUI8L4fJboshUdKXwHog0dYGITTBChX0wbkeNv97SHKe2L8b7TstUCyDKzwSaZv309Ycf/y8PKb5peFUeP1eBJCPDTujb++WGqwjlWmY28wYRjNjuegl5SE0c5uJXDQVdbf72SU/BGRRND4tU3A1KnQRfzcABLbOfVD09iU/kyOj7Xw6BtaYXpO411HNFsaoTzgIoUI03bUKITkabUGGrpkvKYI76uY4TzJt5JXuCKi/Yh8jyDBZ2QQ== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: fQLmqqjaeoSrGqc7IkKTSfk7/nl9bCdoctFG+/fKQVQmNRLTLiSOiDtDRC0+dMpqHp8ediVHr23r2c6OpmTslQWD0GjS60w3C2Nj6uCht3OD+DzOWKdn5M3DD7uXy3gpvL0Daj0neaIrUlWCeECtmnQKYtyvYfg0E2O/2iU2FPyTqJ9vaiETlU5LkJRDTNyHpPXaAc5MHgcMn2F2wW79w3BIf7TxR3QlVSrUumJOvrbTBY5IzJrvdjzUBq2C63n9QhJyxzaD04nBur9+wFOQbIIIxd4UezAEJdupC9cR1TFzUBg+EEbe0KfU4h+68op2iRzr7fplV+ZHJgjIOMTsKB0syIWFtlkqiAQg8iORIdtalMbJKAMGCDVCTLWrDb0wpj2cZ1ilq8DWklarvPxC9jYDTj9O/JEatPZFQE03aoX73AN+27dHX0SsLs9zgDH5T73bTCYCQiw2JYyk6q4d78qtXZ1jKT0eNN40maI+Bjzt2bbovDiYxamyJ20IchdZohtpLk1SdFcnNgl5RK+azDdxFjmi2W9uBk+g1kisDjiFbsKol82SCGPbBe42pRj9MIcVjIY42VMyv5H2uvBjm+v/ifw5nEeNP5T5yqUjQnI= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ff49a02d-8832-489c-b199-08dd5c9a7080 X-MS-Exchange-CrossTenant-AuthSource: MN2PR10MB4112.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Mar 2025 10:34:14.0658 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: lzgTENKg+QaocPm3lYAVLgk3+y9DPKyz92nbzObZoYBnXDReiD8h9gWs3FgAVP+JzuIOFZWho82BFXmS3peUojPGkni9LOfbJV/xljND/Zg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB4647 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1093,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-03-06_05,2025-03-06_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 adultscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502100000 definitions=main-2503060079 X-Proofpoint-GUID: 0FxVLUWEtqOnwcCOpnFfymdcqYIOtzAQ X-Proofpoint-ORIG-GUID: 0FxVLUWEtqOnwcCOpnFfymdcqYIOtzAQ X-Rspamd-Queue-Id: 5E166180004 X-Stat-Signature: p83jr6miaiipx4fr9nbaubaabyhmqhtt X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1741257259-186521 X-HE-Meta: U2FsdGVkX1/A1+ivIKSwYL4vFxio+RBMGXas+udMJHYTX9cDL/SwbGnxkeUf4y94F/yjrRFQKgSxlKDWHCtXmcO2BDRQN/kqPNVvYFPnnlWEVgY5Qib1AQWX3M8G8O1hB4Yn6I3P63Yq5Q5CRxDncAAXz95bdmg1NxdlBsZFhNKAsoeyxdURtq4kb0htSBRVJ6+AqsJhcWaZx1YK0jQUfJZgzY5G5U+pOA6sIT6ZFreP5ESA7pMrHJNS+75lLCR1zWckrCDj4Ieo3ijSCwCib/eSxnnTX5x2TMRvw2aweLa7dqHPKQ2kk5/0ss5B5SJLvl0klqFflZeArhqa02/LXFi49OUsgdEAiUI08ZwiIL7OsYkbzRi1le7Wk6UZt664lg+6pfoIseT+QJIGKYbBhj3+8LYxwx5q/TAiOZivffUlr25xHE2nXCLmArR8W/JNwXnSKQmkrI9JxYTqcahv0Md9xklZfjmZI76wH+pM+Lrbmo7gNA1X/k20LPBPsWAKV0arhRVE7EARqc7ebiMBF82FdleK7I2z+DZBcsK390wcdTGJcRv+P/WdYHl9PhPjGq5TwjcP3MffV5YsfnUON8WmkkiMRqlR179QkMo9RLsaivc3TJopWG/wTyBGqdE8Q7/Q1rv5DT3f3pbNnkzRrS2auey9qh3dLTimC0Hv36d2KBz1n9eDqNDspyUxquroEVbgHBpf7nO6DnjazjhHYeQcd4/+Rx9p1qGyaZe6B6X1X6/mr/tG/F11YUmVjDj0rJ+L4Xz/86AObp8ZaZUBLCCSCR1EhzjZNC0hFjDJ4Yh0YadmhNj7WhQehNqUPQ1tA6V16tD6LEBZE6xRZj5A5m3qpYrBLXxUKgQiM+ZxWq2NHUunjebpE45G4WMh9+bDwX21VvTA7ThkRovYJaOTXa7AO7p0l8p7CKWcW8ushgZHiG9qoyOPEAkmYYfvXfpbv0MaiSd20qe8Vvpd0yY mBENwv1f GX4Ce/WJzmPyYuWLD3iQE55xBBbLMqPKC7XwjVSkDPmd+MKFQH2x48SLRjTMOCw6HgrJt+LSNpgaY4O3eQk/vh5FzlBGteEea3dNDDTEa/h+aQrnPm8UJmq/FuAqO1JoXE99jw9AQ3OIUJW9jrHOEescusleZhD0c94vc+1NfmJyms6xumI/rVjqXtA8q2OSmNM3NvPSUor8vksJX/yI0w4irxy0vFu2/8RpMISjqGWSFjg6cAbYRZ3EltkavhAgol1JqqTzAdH0ZNIojtyEEB/jfYewxvg/lgOGCbKPJSShAcV6b4c3ghuz0z37SEoJaDHDUsqIJelqo+ffdk85/p1DWrBsJm/O+3YNDcC9iRx1CXTiKbZ96TbFbCY+RKm2SWxpJh66jJi/4u5EgOgiLUkzZJhSdIQY9yFxri4U/fLsYfT64AxLo/W5tS5mGsYGTqiFDiXaHPFQ5yhzXSSp1mJamgksbUkEdoOcdZgLp++Vw8F251GOGHAar76bQyA7LmbCWBoD+odDC2AKMcDsXt/6sXUZrprEZrxWAwNzs8JzXZ89vRHUv3SqgG4LT+3GOZoxHb0axVRGC+qCuJn7IuRzckBmPel6ngAbi/JsdgTrlW0lZt3PuZ74HahfjqPs8jt75C8QKNg7Cha4RTuM99wWqGwYB5bch2ikbchXCbVc8Ckoa2M/c7I/N6dA5eeHhx1qE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A number of mremap() calls both pass around and modify a large number of parameters, making the code less readable and often repeatedly having to determine things such as VMA, size delta, and more. Avoid this by using the common pattern of passing a state object through the operation, updating it as we go. We introduce the vma_remap_struct or 'VRM' for this purpose. This also gives us the ability to accumulate further state through the operation that would otherwise require awkward and error-prone pointer passing. We can also now trivially define helper functions that operate on a VRM object. This pattern has proven itself to be very powerful when implemented for VMA merge, VMA unmapping and memory mapping operations, so it is battle-tested and functional. We both introduce the data structure and use it, introducing helper functions as needed to make things readable, we move some state such as mmap lock and mlock() status to the VRM, we introduce a means of classifying the type of mremap() operation and de-duplicate the get_unmapped_area() lookup. We also neatly thread userfaultfd state throughout the operation. Note that there is further refactoring to be done, chiefly adjust move_vma() to accept a VRM parameter. We defer this as there is pre-requisite work required to be able to do so which we will do in a subsequent patch. Signed-off-by: Lorenzo Stoakes Reviewed-by: Vlastimil Babka Signed-off-by: Lorenzo Stoakes Reported-by: kernel test robot --- mm/mremap.c | 565 +++++++++++++++++++++++++++++++++------------------- 1 file changed, 358 insertions(+), 207 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index c4abda8dfc57..0ab0c88072a0 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -32,6 +32,44 @@ #include "internal.h" +/* Classify the kind of remap operation being performed. */ +enum mremap_type { + MREMAP_INVALID, /* Initial state. */ + MREMAP_NO_RESIZE, /* old_len == new_len, if not moved, do nothing. */ + MREMAP_SHRINK, /* old_len > new_len. */ + MREMAP_EXPAND, /* old_len < new_len. */ +}; + +/* + * Describes a VMA mremap() operation and is threaded throughout it. + * + * Any of the fields may be mutated by the operation, however these values will + * always accurately reflect the remap (for instance, we may adjust lengths and + * delta to account for hugetlb alignment). + */ +struct vma_remap_struct { + /* User-provided state. */ + unsigned long addr; /* User-specified address from which we remap. */ + unsigned long old_len; /* Length of range being remapped. */ + unsigned long new_len; /* Desired new length of mapping. */ + unsigned long flags; /* user-specified MREMAP_* flags. */ + unsigned long new_addr; /* Optionally, desired new address. */ + + /* uffd state. */ + struct vm_userfaultfd_ctx *uf; + struct list_head *uf_unmap_early; + struct list_head *uf_unmap; + + /* VMA state, determined in do_mremap(). */ + struct vm_area_struct *vma; + + /* Internal state, determined in do_mremap(). */ + unsigned long delta; /* Absolute delta of old_len,new_len. */ + bool mlocked; /* Was the VMA mlock()'d? */ + enum mremap_type remap_type; /* expand, shrink, etc. */ + bool mmap_locked; /* Is mm currently write-locked? */ +}; + static pud_t *get_old_pud(struct mm_struct *mm, unsigned long addr) { pgd_t *pgd; @@ -693,10 +731,95 @@ unsigned long move_page_tables(struct vm_area_struct *vma, return len + old_addr - old_end; /* how much done */ } +/* Set vrm->delta to the difference in VMA size specified by user. */ +static void vrm_set_delta(struct vma_remap_struct *vrm) +{ + vrm->delta = abs_diff(vrm->old_len, vrm->new_len); +} + +/* Determine what kind of remap this is - shrink, expand or no resize at all. */ +static enum mremap_type vrm_remap_type(struct vma_remap_struct *vrm) +{ + if (vrm->delta == 0) + return MREMAP_NO_RESIZE; + + if (vrm->old_len > vrm->new_len) + return MREMAP_SHRINK; + + return MREMAP_EXPAND; +} + +/* + * When moving a VMA to vrm->new_adr, does this result in the new and old VMAs + * overlapping? + */ +static bool vrm_overlaps(struct vma_remap_struct *vrm) +{ + unsigned long start_old = vrm->addr; + unsigned long start_new = vrm->new_addr; + unsigned long end_old = vrm->addr + vrm->old_len; + unsigned long end_new = vrm->new_addr + vrm->new_len; + + /* + * start_old end_old + * |-----------| + * | | + * |-----------| + * |-------------| + * | | + * |-------------| + * start_new end_new + */ + if (end_old > start_new && end_new > start_old) + return true; + + return false; +} + +/* Do the mremap() flags require that the new_addr parameter be specified? */ +static bool vrm_implies_new_addr(struct vma_remap_struct *vrm) +{ + return vrm->flags & (MREMAP_FIXED | MREMAP_DONTUNMAP); +} + +/* + * Find an unmapped area for the requested vrm->new_addr. + * + * If MREMAP_FIXED then this is equivalent to a MAP_FIXED mmap() call. If only + * MREMAP_DONTUNMAP is set, then this is equivalent to providing a hint to + * mmap(), otherwise this is equivalent to mmap() specifying a NULL address. + * + * Returns 0 on success (with vrm->new_addr updated), or an error code upon + * failure. + */ +static unsigned long vrm_set_new_addr(struct vma_remap_struct *vrm) +{ + struct vm_area_struct *vma = vrm->vma; + unsigned long map_flags = 0; + /* Page Offset _into_ the VMA. */ + pgoff_t internal_pgoff = (vrm->addr - vma->vm_start) >> PAGE_SHIFT; + pgoff_t pgoff = vma->vm_pgoff + internal_pgoff; + unsigned long new_addr = vrm_implies_new_addr(vrm) ? vrm->new_addr : 0; + unsigned long res; + + if (vrm->flags & MREMAP_FIXED) + map_flags |= MAP_FIXED; + if (vma->vm_flags & VM_MAYSHARE) + map_flags |= MAP_SHARED; + + res = get_unmapped_area(vma->vm_file, new_addr, vrm->new_len, pgoff, + map_flags); + if (IS_ERR_VALUE(res)) + return res; + + vrm->new_addr = res; + return 0; +} + static unsigned long move_vma(struct vm_area_struct *vma, unsigned long old_addr, unsigned long old_len, unsigned long new_len, unsigned long new_addr, - bool *locked, unsigned long flags, + bool *mlocked, unsigned long flags, struct vm_userfaultfd_ctx *uf, struct list_head *uf_unmap) { long to_account = new_len - old_len; @@ -837,7 +960,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, if (vm_flags & VM_LOCKED) { mm->locked_vm += new_len >> PAGE_SHIFT; - *locked = true; + *mlocked = true; } mm->hiwater_vm = hiwater_vm; @@ -860,18 +983,15 @@ static unsigned long move_vma(struct vm_area_struct *vma, * resize_is_valid() - Ensure the vma can be resized to the new length at the give * address. * - * @vma: The vma to resize - * @addr: The old address - * @old_len: The current size - * @new_len: The desired size - * @flags: The vma flags - * * Return 0 on success, error otherwise. */ -static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr, - unsigned long old_len, unsigned long new_len, unsigned long flags) +static int resize_is_valid(struct vma_remap_struct *vrm) { struct mm_struct *mm = current->mm; + struct vm_area_struct *vma = vrm->vma; + unsigned long addr = vrm->addr; + unsigned long old_len = vrm->old_len; + unsigned long new_len = vrm->new_len; unsigned long pgoff; /* @@ -883,11 +1003,12 @@ static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr, * behavior. As a result, fail such attempts. */ if (!old_len && !(vma->vm_flags & (VM_SHARED | VM_MAYSHARE))) { - pr_warn_once("%s (%d): attempted to duplicate a private mapping with mremap. This is not supported.\n", current->comm, current->pid); + pr_warn_once("%s (%d): attempted to duplicate a private mapping with mremap. This is not supported.\n", + current->comm, current->pid); return -EINVAL; } - if ((flags & MREMAP_DONTUNMAP) && + if ((vrm->flags & MREMAP_DONTUNMAP) && (vma->vm_flags & (VM_DONTEXPAND | VM_PFNMAP))) return -EINVAL; @@ -907,99 +1028,117 @@ static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr, if (vma->vm_flags & (VM_DONTEXPAND | VM_PFNMAP)) return -EFAULT; - if (!mlock_future_ok(mm, vma->vm_flags, new_len - old_len)) + if (!mlock_future_ok(mm, vma->vm_flags, vrm->delta)) return -EAGAIN; - if (!may_expand_vm(mm, vma->vm_flags, - (new_len - old_len) >> PAGE_SHIFT)) + if (!may_expand_vm(mm, vma->vm_flags, vrm->delta >> PAGE_SHIFT)) return -ENOMEM; return 0; } /* - * mremap_to() - remap a vma to a new location - * @addr: The old address - * @old_len: The old size - * @new_addr: The target address - * @new_len: The new size - * @locked: If the returned vma is locked (VM_LOCKED) - * @flags: the mremap flags - * @uf: The mremap userfaultfd context - * @uf_unmap_early: The userfaultfd unmap early context - * @uf_unmap: The userfaultfd unmap context + * The user has requested that the VMA be shrunk (i.e., old_len > new_len), so + * execute this, optionally dropping the mmap lock when we do so. * + * In both cases this invalidates the VMA, however if we don't drop the lock, + * then load the correct VMA into vrm->vma afterwards. + */ +static unsigned long shrink_vma(struct vma_remap_struct *vrm, + bool drop_lock) +{ + struct mm_struct *mm = current->mm; + unsigned long unmap_start = vrm->addr + vrm->new_len; + unsigned long unmap_bytes = vrm->delta; + unsigned long res; + VMA_ITERATOR(vmi, mm, unmap_start); + + VM_BUG_ON(vrm->remap_type != MREMAP_SHRINK); + + res = do_vmi_munmap(&vmi, mm, unmap_start, unmap_bytes, + vrm->uf_unmap, drop_lock); + vrm->vma = NULL; /* Invalidated. */ + if (res) + return res; + + /* + * If we've not dropped the lock, then we should reload the VMA to + * replace the invalidated VMA with the one that may have now been + * split. + */ + if (drop_lock) + vrm->mmap_locked = false; + else + vrm->vma = vma_lookup(mm, vrm->addr); + + return 0; +} + +/* + * mremap_to() - remap a vma to a new location. * Returns: The new address of the vma or an error. */ -static unsigned long mremap_to(unsigned long addr, unsigned long old_len, - unsigned long new_addr, unsigned long new_len, bool *locked, - unsigned long flags, struct vm_userfaultfd_ctx *uf, - struct list_head *uf_unmap_early, - struct list_head *uf_unmap) +static unsigned long mremap_to(struct vma_remap_struct *vrm) { struct mm_struct *mm = current->mm; - struct vm_area_struct *vma; - unsigned long ret; - unsigned long map_flags = 0; + unsigned long err; /* Is the new length or address silly? */ - if (new_len > TASK_SIZE || new_addr > TASK_SIZE - new_len) + if (vrm->new_len > TASK_SIZE || + vrm->new_addr > TASK_SIZE - vrm->new_len) return -EINVAL; - /* Ensure the old/new locations do not overlap. */ - if (addr + old_len > new_addr && new_addr + new_len > addr) + if (vrm_overlaps(vrm)) return -EINVAL; - if (flags & MREMAP_FIXED) { + if (vrm->flags & MREMAP_FIXED) { /* * In mremap_to(). * VMA is moved to dst address, and munmap dst first. * do_munmap will check if dst is sealed. */ - ret = do_munmap(mm, new_addr, new_len, uf_unmap_early); - if (ret) - return ret; - } + err = do_munmap(mm, vrm->new_addr, vrm->new_len, + vrm->uf_unmap_early); + vrm->vma = NULL; /* Invalidated. */ + if (err) + return err; - if (old_len > new_len) { - ret = do_munmap(mm, addr+new_len, old_len - new_len, uf_unmap); - if (ret) - return ret; - old_len = new_len; + /* + * If we remap a portion of a VMA elsewhere in the same VMA, + * this can invalidate the old VMA. Reset. + */ + vrm->vma = vma_lookup(mm, vrm->addr); } - vma = vma_lookup(mm, addr); - if (!vma) - return -EFAULT; - - ret = resize_is_valid(vma, addr, old_len, new_len, flags); - if (ret) - return ret; + if (vrm->remap_type == MREMAP_SHRINK) { + err = shrink_vma(vrm, /* drop_lock= */false); + if (err) + return err; - /* MREMAP_DONTUNMAP expands by old_len since old_len == new_len */ - if (flags & MREMAP_DONTUNMAP && - !may_expand_vm(mm, vma->vm_flags, old_len >> PAGE_SHIFT)) { - return -ENOMEM; + /* Set up for the move now shrink has been executed. */ + vrm->old_len = vrm->new_len; } - if (flags & MREMAP_FIXED) - map_flags |= MAP_FIXED; + err = resize_is_valid(vrm); + if (err) + return err; - if (vma->vm_flags & VM_MAYSHARE) - map_flags |= MAP_SHARED; + /* MREMAP_DONTUNMAP expands by old_len since old_len == new_len */ + if (vrm->flags & MREMAP_DONTUNMAP) { + vm_flags_t vm_flags = vrm->vma->vm_flags; + unsigned long pages = vrm->old_len >> PAGE_SHIFT; - ret = get_unmapped_area(vma->vm_file, new_addr, new_len, vma->vm_pgoff + - ((addr - vma->vm_start) >> PAGE_SHIFT), - map_flags); - if (IS_ERR_VALUE(ret)) - return ret; + if (!may_expand_vm(mm, vm_flags, pages)) + return -ENOMEM; + } - /* We got a new mapping */ - if (!(flags & MREMAP_FIXED)) - new_addr = ret; + err = vrm_set_new_addr(vrm); + if (err) + return err; - return move_vma(vma, addr, old_len, new_len, new_addr, locked, flags, - uf, uf_unmap); + return move_vma(vrm->vma, vrm->addr, vrm->old_len, vrm->new_len, + vrm->new_addr, &vrm->mlocked, vrm->flags, + vrm->uf, vrm->uf_unmap); } static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) @@ -1016,22 +1155,33 @@ static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) return 1; } -/* Do the mremap() flags require that the new_addr parameter be specified? */ -static bool implies_new_addr(unsigned long flags) +/* Determine whether we are actually able to execute an in-place expansion. */ +static bool vrm_can_expand_in_place(struct vma_remap_struct *vrm) { - return flags & (MREMAP_FIXED | MREMAP_DONTUNMAP); + /* Number of bytes from vrm->addr to end of VMA. */ + unsigned long suffix_bytes = vrm->vma->vm_end - vrm->addr; + + /* If end of range aligns to end of VMA, we can just expand in-place. */ + if (suffix_bytes != vrm->old_len) + return false; + + /* Check whether this is feasible. */ + if (!vma_expandable(vrm->vma, vrm->delta)) + return false; + + return true; } /* * Are the parameters passed to mremap() valid? If so return 0, otherwise return * error. */ -static unsigned long check_mremap_params(unsigned long addr, - unsigned long flags, - unsigned long old_len, - unsigned long new_len, - unsigned long new_addr) +static unsigned long check_mremap_params(struct vma_remap_struct *vrm) + { + unsigned long addr = vrm->addr; + unsigned long flags = vrm->flags; + /* Ensure no unexpected flag values. */ if (flags & ~(MREMAP_FIXED | MREMAP_MAYMOVE | MREMAP_DONTUNMAP)) return -EINVAL; @@ -1045,15 +1195,15 @@ static unsigned long check_mremap_params(unsigned long addr, * for DOS-emu "duplicate shm area" thing. But * a zero new-len is nonsensical. */ - if (!PAGE_ALIGN(new_len)) + if (!PAGE_ALIGN(vrm->new_len)) return -EINVAL; /* Remainder of checks are for cases with specific new_addr. */ - if (!implies_new_addr(flags)) + if (!vrm_implies_new_addr(vrm)) return 0; /* The new address must be page-aligned. */ - if (offset_in_page(new_addr)) + if (offset_in_page(vrm->new_addr)) return -EINVAL; /* A fixed address implies a move. */ @@ -1061,7 +1211,7 @@ static unsigned long check_mremap_params(unsigned long addr, return -EINVAL; /* MREMAP_DONTUNMAP does not allow resizing in the process. */ - if (flags & MREMAP_DONTUNMAP && old_len != new_len) + if (flags & MREMAP_DONTUNMAP && vrm->old_len != vrm->new_len) return -EINVAL; /* @@ -1090,11 +1240,11 @@ static unsigned long check_mremap_params(unsigned long addr, * If we discover the VMA is locked, update mm_struct statistics accordingly and * indicate so to the caller. */ -static unsigned long expand_vma_inplace(struct vm_area_struct *vma, - unsigned long delta, bool *locked) +static unsigned long expand_vma_in_place(struct vma_remap_struct *vrm) { struct mm_struct *mm = current->mm; - long pages = delta >> PAGE_SHIFT; + long pages = vrm->delta >> PAGE_SHIFT; + struct vm_area_struct *vma = vrm->vma; VMA_ITERATOR(vmi, mm, vma->vm_end); long charged = 0; @@ -1114,7 +1264,7 @@ static unsigned long expand_vma_inplace(struct vm_area_struct *vma, * adjacent to the expanded vma and otherwise * compatible. */ - vma = vma_merge_extend(&vmi, vma, delta); + vma = vrm->vma = vma_merge_extend(&vmi, vma, vrm->delta); if (!vma) { vm_unacct_memory(charged); return -ENOMEM; @@ -1123,42 +1273,34 @@ static unsigned long expand_vma_inplace(struct vm_area_struct *vma, vm_stat_account(mm, vma->vm_flags, pages); if (vma->vm_flags & VM_LOCKED) { mm->locked_vm += pages; - *locked = true; + vrm->mlocked = true; } return 0; } -static bool align_hugetlb(struct vm_area_struct *vma, - unsigned long addr, - unsigned long new_addr, - unsigned long *old_len_ptr, - unsigned long *new_len_ptr, - unsigned long *delta_ptr) +static bool align_hugetlb(struct vma_remap_struct *vrm) { - unsigned long old_len = *old_len_ptr; - unsigned long new_len = *new_len_ptr; - struct hstate *h __maybe_unused = hstate_vma(vma); + struct hstate *h __maybe_unused = hstate_vma(vrm->vma); - old_len = ALIGN(old_len, huge_page_size(h)); - new_len = ALIGN(new_len, huge_page_size(h)); + vrm->old_len = ALIGN(vrm->old_len, huge_page_size(h)); + vrm->new_len = ALIGN(vrm->new_len, huge_page_size(h)); /* addrs must be huge page aligned */ - if (addr & ~huge_page_mask(h)) + if (vrm->addr & ~huge_page_mask(h)) return false; - if (new_addr & ~huge_page_mask(h)) + if (vrm->new_addr & ~huge_page_mask(h)) return false; /* * Don't allow remap expansion, because the underlying hugetlb * reservation is not yet capable to handle split reservation. */ - if (new_len > old_len) + if (vrm->new_len > vrm->old_len) return false; - *old_len_ptr = old_len; - *new_len_ptr = new_len; - *delta_ptr = abs_diff(old_len, new_len); + vrm_set_delta(vrm); + return true; } @@ -1169,19 +1311,16 @@ static bool align_hugetlb(struct vm_area_struct *vma, * Try to do so in-place, if this fails, then move the VMA to a new location to * action the change. */ -static unsigned long expand_vma(struct vm_area_struct *vma, - unsigned long addr, unsigned long old_len, - unsigned long new_len, unsigned long flags, - bool *locked_ptr, unsigned long *new_addr_ptr, - struct vm_userfaultfd_ctx *uf_ptr, - struct list_head *uf_unmap_ptr) +static unsigned long expand_vma(struct vma_remap_struct *vrm) { unsigned long err; - unsigned long map_flags; - unsigned long new_addr; /* We ignore any user-supplied one. */ - pgoff_t pgoff; + struct vm_area_struct *vma = vrm->vma; + unsigned long addr = vrm->addr; + unsigned long old_len = vrm->old_len; + unsigned long new_len = vrm->new_len; + unsigned long flags = vrm->flags; - err = resize_is_valid(vma, addr, old_len, new_len, flags); + err = resize_is_valid(vrm); if (err) return err; @@ -1189,10 +1328,9 @@ static unsigned long expand_vma(struct vm_area_struct *vma, * [addr, old_len) spans precisely to the end of the VMA, so try to * expand it in-place. */ - if (old_len == vma->vm_end - addr && - vma_expandable(vma, new_len - old_len)) { - err = expand_vma_inplace(vma, new_len - old_len, locked_ptr); - if (IS_ERR_VALUE(err)) + if (vrm_can_expand_in_place(vrm)) { + err = expand_vma_in_place(vrm); + if (err) return err; /* @@ -1200,8 +1338,8 @@ static unsigned long expand_vma(struct vm_area_struct *vma, * satisfy the expectation that mlock()'ing a VMA maintains all * of its pages in memory. */ - if (*locked_ptr) - *new_addr_ptr = addr; + if (vrm->mlocked) + vrm->new_addr = addr; /* OK we're done! */ return addr; @@ -1217,62 +1355,67 @@ static unsigned long expand_vma(struct vm_area_struct *vma, return -ENOMEM; /* Find a new location to move the VMA to. */ - map_flags = (vma->vm_flags & VM_MAYSHARE) ? MAP_SHARED : 0; - pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >> PAGE_SHIFT); - new_addr = get_unmapped_area(vma->vm_file, 0, new_len, pgoff, map_flags); - if (IS_ERR_VALUE(new_addr)) - return new_addr; - *new_addr_ptr = new_addr; + err = vrm_set_new_addr(vrm); + if (err) + return err; - return move_vma(vma, addr, old_len, new_len, new_addr, - locked_ptr, flags, uf_ptr, uf_unmap_ptr); + return move_vma(vma, addr, old_len, new_len, vrm->new_addr, + &vrm->mlocked, flags, vrm->uf, vrm->uf_unmap); } /* - * Expand (or shrink) an existing mapping, potentially moving it at the - * same time (controlled by the MREMAP_MAYMOVE flag and available VM space) - * - * MREMAP_FIXED option added 5-Dec-1999 by Benjamin LaHaise - * This option implies MREMAP_MAYMOVE. + * Attempt to resize the VMA in-place, if we cannot, then move the VMA to the + * first available address to perform the operation. */ -SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, - unsigned long, new_len, unsigned long, flags, - unsigned long, new_addr) +static unsigned long mremap_at(struct vma_remap_struct *vrm) +{ + unsigned long res; + + switch (vrm->remap_type) { + case MREMAP_INVALID: + break; + case MREMAP_NO_RESIZE: + /* NO-OP CASE - resizing to the same size. */ + return vrm->addr; + case MREMAP_SHRINK: + /* + * SHRINK CASE. Can always be done in-place. + * + * Simply unmap the shrunken portion of the VMA. This does all + * the needed commit accounting, and we indicate that the mmap + * lock should be dropped. + */ + res = shrink_vma(vrm, /* drop_lock= */true); + if (res) + return res; + + return vrm->addr; + case MREMAP_EXPAND: + return expand_vma(vrm); + } + + BUG(); +} + +static unsigned long do_mremap(struct vma_remap_struct *vrm) { struct mm_struct *mm = current->mm; struct vm_area_struct *vma; unsigned long ret; - unsigned long delta; - bool locked = false; - struct vm_userfaultfd_ctx uf = NULL_VM_UFFD_CTX; - LIST_HEAD(uf_unmap_early); - LIST_HEAD(uf_unmap); - /* - * There is a deliberate asymmetry here: we strip the pointer tag - * from the old address but leave the new address alone. This is - * for consistency with mmap(), where we prevent the creation of - * aliasing mappings in userspace by leaving the tag bits of the - * mapping address intact. A non-zero tag will cause the subsequent - * range checks to reject the address as invalid. - * - * See Documentation/arch/arm64/tagged-address-abi.rst for more - * information. - */ - addr = untagged_addr(addr); - - ret = check_mremap_params(addr, flags, old_len, new_len, new_addr); + ret = check_mremap_params(vrm); if (ret) return ret; - old_len = PAGE_ALIGN(old_len); - new_len = PAGE_ALIGN(new_len); - delta = abs_diff(old_len, new_len); + vrm->old_len = PAGE_ALIGN(vrm->old_len); + vrm->new_len = PAGE_ALIGN(vrm->new_len); + vrm_set_delta(vrm); if (mmap_write_lock_killable(mm)) return -EINTR; + vrm->mmap_locked = true; - vma = vma_lookup(mm, addr); + vma = vrm->vma = vma_lookup(mm, vrm->addr); if (!vma) { ret = -EFAULT; goto out; @@ -1285,62 +1428,70 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, } /* Align to hugetlb page size, if required. */ - if (is_vm_hugetlb_page(vma) && - !align_hugetlb(vma, addr, new_addr, &old_len, &new_len, &delta)) { + if (is_vm_hugetlb_page(vma) && !align_hugetlb(vrm)) { ret = -EINVAL; goto out; } - /* Are we RELOCATING the VMA to a SPECIFIC address? */ - if (implies_new_addr(flags)) { - ret = mremap_to(addr, old_len, new_addr, new_len, - &locked, flags, &uf, &uf_unmap_early, - &uf_unmap); - goto out; - } - - /* - * From here on in we are only RESIZING the VMA, attempting to do so - * in-place, moving the VMA if we cannot. - */ + vrm->remap_type = vrm_remap_type(vrm); - /* NO-OP CASE - resizing to the same size. */ - if (new_len == old_len) { - ret = addr; - goto out; - } + /* Actually execute mremap. */ + ret = vrm_implies_new_addr(vrm) ? mremap_to(vrm) : mremap_at(vrm); - /* SHRINK CASE. Can always be done in-place. */ - if (new_len < old_len) { - VMA_ITERATOR(vmi, mm, addr + new_len); +out: + if (vrm->mmap_locked) { + mmap_write_unlock(mm); + vrm->mmap_locked = false; - /* - * Simply unmap the shrunken portion of the VMA. This does all - * the needed commit accounting, unlocking the mmap lock. - */ - ret = do_vmi_munmap(&vmi, mm, addr + new_len, delta, - &uf_unmap, true); - if (ret) - goto out; - - /* We succeeded, mmap lock released for us. */ - ret = addr; - goto out_unlocked; + if (!offset_in_page(ret) && vrm->mlocked && vrm->new_len > vrm->old_len) + mm_populate(vrm->new_addr + vrm->old_len, vrm->delta); } - /* EXPAND case. We try to do in-place, if we can't, then we move it. */ - ret = expand_vma(vma, addr, old_len, new_len, flags, &locked, &new_addr, - &uf, &uf_unmap); + userfaultfd_unmap_complete(mm, vrm->uf_unmap_early); + mremap_userfaultfd_complete(vrm->uf, vrm->addr, ret, vrm->old_len); + userfaultfd_unmap_complete(mm, vrm->uf_unmap); -out: - if (offset_in_page(ret)) - locked = false; - mmap_write_unlock(mm); - if (locked && new_len > old_len) - mm_populate(new_addr + old_len, delta); -out_unlocked: - userfaultfd_unmap_complete(mm, &uf_unmap_early); - mremap_userfaultfd_complete(&uf, addr, ret, old_len); - userfaultfd_unmap_complete(mm, &uf_unmap); return ret; } + +/* + * Expand (or shrink) an existing mapping, potentially moving it at the + * same time (controlled by the MREMAP_MAYMOVE flag and available VM space) + * + * MREMAP_FIXED option added 5-Dec-1999 by Benjamin LaHaise + * This option implies MREMAP_MAYMOVE. + */ +SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, + unsigned long, new_len, unsigned long, flags, + unsigned long, new_addr) +{ + struct vm_userfaultfd_ctx uf = NULL_VM_UFFD_CTX; + LIST_HEAD(uf_unmap_early); + LIST_HEAD(uf_unmap); + /* + * There is a deliberate asymmetry here: we strip the pointer tag + * from the old address but leave the new address alone. This is + * for consistency with mmap(), where we prevent the creation of + * aliasing mappings in userspace by leaving the tag bits of the + * mapping address intact. A non-zero tag will cause the subsequent + * range checks to reject the address as invalid. + * + * See Documentation/arch/arm64/tagged-address-abi.rst for more + * information. + */ + struct vma_remap_struct vrm = { + .addr = untagged_addr(addr), + .old_len = old_len, + .new_len = new_len, + .flags = flags, + .new_addr = new_addr, + + .uf = &uf, + .uf_unmap_early = &uf_unmap_early, + .uf_unmap = &uf_unmap, + + .remap_type = MREMAP_INVALID, /* We set later. */ + }; + + return do_mremap(&vrm); +} From patchwork Thu Mar 6 10:34:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Stoakes X-Patchwork-Id: 14004151 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA87EC282EC for ; Thu, 6 Mar 2025 10:34:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A21A5280006; Thu, 6 Mar 2025 05:34:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9CD16280003; Thu, 6 Mar 2025 05:34:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7858D280006; Thu, 6 Mar 2025 05:34:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 54E10280003 for ; Thu, 6 Mar 2025 05:34:23 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C332EA9DD1 for ; Thu, 6 Mar 2025 10:34:23 +0000 (UTC) X-FDA: 83190766806.11.A2C3EFA Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf29.hostedemail.com (Postfix) with ESMTP id 8F019120009 for ; Thu, 6 Mar 2025 10:34:20 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=Q3gioSBF; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=WKwHkTcU; spf=pass (imf29.hostedemail.com: domain of lorenzo.stoakes@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=lorenzo.stoakes@oracle.com; dmarc=pass (policy=reject) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741257260; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j98RvGp9i8dPsGlpfoEaNFCEMACsgE+Sk7Yql+1C7n8=; b=XlyJUbZs/iwgbSYu1SbkbWtdi1vKfmAY9yGDBlwjHRyPJE9qOPppjKyBsDRLpxg4/Pz20Q L+T902Txnxd5HbviKM3jVmbgxdhcjFkNb6qKtvIQAHhk3w0m9TMJeVpTx16tnxoYtwsmTU gyahx7c04nBb8NZRWsd7N8PZgwtZJ98= ARC-Authentication-Results: i=2; imf29.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=Q3gioSBF; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=WKwHkTcU; spf=pass (imf29.hostedemail.com: domain of lorenzo.stoakes@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=lorenzo.stoakes@oracle.com; dmarc=pass (policy=reject) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1741257260; a=rsa-sha256; cv=pass; b=FA5K1caB4iDUiHFLFRfe7iwiWJxfpfdrDjBc0hz4QC3BgJqMcryHNiIxPX1VzdMNlWQSP0 DlDgNDc91D3bEcDCqQ7fvHT0b3zn3zntgJefSw8IA9XDs5IfH8DzoUNTSY+uX4EprVat+l agif31WO6GRChiNJRiBzfU2nR3zJR/0= Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 526ABflb003723; Thu, 6 Mar 2025 10:34:18 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=j98RvGp9i8dPsGlpfoEaNFCEMACsgE+Sk7Yql+1C7n8=; b= Q3gioSBF34FlXIsgmUjS3+B0PAToBfA69z+Ue8DeYcTyz+n0rcP3JJEUOZy8/EFZ s4mOR240Byo73XUyGLbvJJnEqM68pVKaXHAtbUiVILCToyw7leSKfLFz1hvwAtDb ELJsxp4b9ctBG+T/GQz21wVDKqY/xx743R7zqaLXdW0RGmxNe9qRU3yzeYSYF2wU 9PVx9OF4vfB4OIT1nGAITX1z8H1FFdFCppBqSyZ+6HiWfjWnUGLIg+PGEH2EAK0e WrV+uouTNRJhWNMGjXS13tk9yUDP4yKPwkcTLUwLHfzYCQ5i97jaRsGN74ukn4sL copudIJqSTbSMow2457W+Q== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 453u8wspbx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 06 Mar 2025 10:34:18 +0000 (GMT) Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 5268XesT010933; Thu, 6 Mar 2025 10:34:17 GMT Received: from nam04-bn8-obe.outbound.protection.outlook.com (mail-bn8nam04lp2049.outbound.protection.outlook.com [104.47.74.49]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 453rpdh6rw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 06 Mar 2025 10:34:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=g2vWNLytX8iY3MsrST/sJnS+5aDholziyTOfbTGc8hMB7AiChg0GiSAofjuO+ZUFywttgAoqscZb44Eayh5zg64UXfpHBvIrAYeSLsgjoh4i6u/lsE9iNdy2Im93f4F/jTTw51zEq12AqdZh1ewqTPjxmFsPLpJi1X1aLq8Kxl0ACrn9353FKBzS2hlIxsPs5BfLr6l08pjKJkIVXIegg1nRMArMV3OhPz/Rmv6reY2tkm676ypJFxLw/oHvFQmzmbsurDKIaHmXJ381skpJjcS56CvTjJLHB00jN6Xd0qz6zcXEEwqe4Xw28QB/A+nJ5pCvIz34rkagaoSHclkvvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=j98RvGp9i8dPsGlpfoEaNFCEMACsgE+Sk7Yql+1C7n8=; b=AlhqQlFpYEdpzvCmRVZbjCwxHofUcE7hdK4T3FYNGN1c9xi1Qjp/Vew2jog3MO79znvMRbmuV+/62kCENAG28xtVNqoN88jrBg2U1IvpusP1MdiGAnzJwqzY6cJV3olE7aD9nvWZzh7AuOkGkgYxksej9d+opTgv7OTFHQGIgCdRf5CQFmOb6p3z+i1Ml7KBCfLvb+4yLef6guIFSgbllcWVSZ2sDzpzJr79xAn8JoExZVe9BkJYdbmGZ1e0kfnFxGlvBTPbL0PiC4KJZXVzNIc6mWhcUbdFiWa63PV+tXaeDnAh6tSnIRYA61w6CpQnl+rcuhpQPhnxaDahpyuAMA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=j98RvGp9i8dPsGlpfoEaNFCEMACsgE+Sk7Yql+1C7n8=; b=WKwHkTcUH2dDggkD9ZOb4TDzIggVkRQHSw5eL3HFcX/ZLMYevxv+VJGjiH57Z/Pm5uFs/Dh3uJ6LIv0LFMYaqkzwalSOpo67Rg9GTknL37p6m7EAfiFjRn7/QzBWsL6XTH8VUKADcpii0G+QmXIyPw7e9Jde0KuY0RJDRuD6Wzo= Received: from MN2PR10MB4112.namprd10.prod.outlook.com (2603:10b6:208:11e::33) by PH0PR10MB4647.namprd10.prod.outlook.com (2603:10b6:510:43::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.20; Thu, 6 Mar 2025 10:34:16 +0000 Received: from MN2PR10MB4112.namprd10.prod.outlook.com ([fe80::3256:3c8c:73a9:5b9c]) by MN2PR10MB4112.namprd10.prod.outlook.com ([fe80::3256:3c8c:73a9:5b9c%7]) with mapi id 15.20.8489.025; Thu, 6 Mar 2025 10:34:16 +0000 From: Lorenzo Stoakes To: Andrew Morton Cc: "Liam R . Howlett" , Vlastimil Babka , Jann Horn , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Harry Yoo , Yosry Ahmed Subject: [PATCH v2 4/7] mm/mremap: initial refactor of move_vma() Date: Thu, 6 Mar 2025 10:34:00 +0000 Message-ID: <93193b2b45f6e95da6e8c7a2776e2d8c43a346e7.1741256580.git.lorenzo.stoakes@oracle.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: X-ClientProxiedBy: LO4P123CA0580.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:276::16) To MN2PR10MB4112.namprd10.prod.outlook.com (2603:10b6:208:11e::33) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN2PR10MB4112:EE_|PH0PR10MB4647:EE_ X-MS-Office365-Filtering-Correlation-Id: 0a6e25a1-1a35-4a0e-05e1-08dd5c9a71be X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|366016; X-Microsoft-Antispam-Message-Info: fwck/pNZxpeBbBW5Zmky0xMSiFb9wc3/kmO66QrWLsfSN3egNvmh/mfiGUx5U/HPjrRibIkg86b4H+tetqyK1xBXhvxb2WenXsihPAuwWXfFPMalhT6pBnjWlWYB9nyhsjIApBWj1Uh+fxsceqbbwzb3Xsugh/GWJtmAmO7tXAATwV2wKIPmByoHfF87NsZp5kMc+bMaINmYeRAWTk5WJkuL3r+R+Wc7OjI2FuaBUYo7y1uYP/7x4s8WYtmOJJe41vqEq7nJqarI7354gpPlG3H3NiNY7/eWgGctAvFEnE0C9ebQb0sYe8Nm5I3R4ugQLqrpY/jR06RsGGHQ7LI5gOuBFxS5+iHeNTa+PW4Ox2z3aQcLoev/YFqK/ecorTCNeOV/3oONRKX7hNsmPDNLOUOplSIVA552pr1vdFNeujUJQCz/qAChokIhcL1i8Igv+4Xk79of+jETdJZjoIly0KkDKM5aVESebzeKHWwObJG+jazny79a0eUT4x7NmHZSNF+IwblMJMeF6abry8IP9e5KOuMgA/8RvZaj54CMTWMuwe2dREoB6os8yMm1hG3pOFUvy0cyBA1KiunQ2Jy2yvMX1atvrmTt/HqUp8dakfv6P/vWJDqPIRtsgebBqMQCAwk5L1y6YBVhFAN3W5Aa+G6kviFsOFsCxouM9xUhX9JTLvnWlkFsVxk8EuRZXN8q5M+xzVzbieI6WvrrM/yIFZHY/VNmrbooWkS2hoU52aDoqIJBPOF2Bp8O5EhP7dfwYbDoMbQgnTx1VbJcxxTFXl23DmPPnHPYAkMxR4+qt/pXQB5T8KAu2CH1UqxLTfeixLWJHrYz3l14b86ynVyv9VwINLIers1gx7p7qRSvl1NmyEYGl0IFsL6pPfZqtbQYWnmwE5Jz8hDcY1LlvHTxybkNoz0jgztF5uESvVRCEGJlY5UUkKpCimznPMFgAX+k3K5xEFLb/a/5shtUhZnqsr329C+jg/geFxhuok+rP06VutbYMGd0hAvSX42VgFgxTCUfClaYTy6osyJz1zNksal/r3T/1H1sZtg2Y79sKY0Ov2Ik4ux3Ba+6T33Rstk1lofBEc6QFuts5GEqqdbiTHEf6hIHfYw7LaUdrX4RSLw2fYQ3X/4C1hkhbHb47dHekxxx7uao+vXsWVGQj8jHYlLDA+3tslAp18oFgyKXkPpLcgp3g37ANJ42QuP/2tDIP+y98MMTcLS7mGhaV6pqghDOfu58qQ0IodKLqNbAFl6jTE5GALikLsU70Q4DZfH3IfvIFKrKBI/r2C++o08FEGGiRqedfa9t0ifdFAnMvjYbYRdwyqdsdxq1CGVXZG4Ks/Br/SIox8l3x5LWN7I2DCPBylz77aAg9b4tFwEYCvXGiZ5EHz9pRRyRdP8NkYnc X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MN2PR10MB4112.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: WyUD5NUNm6LTbkZe3Fqnz+Ijifx7R5oJwFfgfXgrEjy104h8/wo3gcpDfqC/8yJTigfkwQsLGeO7U1mAGy7ElNuxKqakUZN6xxKn/joFEBGbrxYSJTLlSHk5Woc6weJR5CmM+W9Vhl599Rc9eoaELj+BVeGU7aT5r6aFfXzjN20DhcnyLumnoN2dFKZ6r/ygWVpyOpaG2EnWqYGeEOyAVgWF3LWye51SigTBoX7j1okSDtu4o5dx3scLp4IhC/+GsU0Wcy4vIOC1bK/bEyE1shcYfOcZ/3gsCLWwaoYJMo+C32gFxU8UhN9WvYK435buNQxqcaVjqgMqWgt3eKqyrixgKT/xvmsSWGP8nskyV52XZMPL+r8+4sZ51XSk+5CBxxof1npV2lFhqYZ0gfpX0pzkH+e0uWngv9ox92UE+OlYpAr1zVBmXJEVttyfCHZM/dxs8yILiI+QWuyYoXvHKz9ahWy5fxdfhHHJg+fpFwn3mXMqglnDNwWlfZxreJQ4vIeGL7PeBwD60QBWpaQY/e6xIj/uJxQ1xajPGUbO30iWeyJMJJxTFslSknV4s+Kgp2Ff6CRVygOQSMsSDzD7Se2lIFNE+YgOxBOpsc5h9i20m+I17wJvuxloLeNgLDcf/o9GyWS5lYacxvIPzAt3mgGSuSQWNV13SQoVNOOVI7wSY0tulYbfsVaOR3cd2xXhzWAYmgNveBrUT/Yl4bTiRsNOBNl4dvHKoZmYIZstFjqBAJWgsWYTyR9AlrdSlFOXQhbnSKVJ8P2P4uvtxemlF6aM6QQ9dU6MsH3/pvI4Ch6MDjqfV/Mz3mC30R3a4KuumXALaWA9HlzKFwFPLN4KDeh8WwKwjxvvFOoupFuc1VpRG/bxt1V3SIsAz7xIqbWxFx9BL9a31YEFaoko5exFLKqJf6O4G8qGq7nz011HZxYXEaq5rJIToTkpuhFlf6gt4+a0K0g9t2IVrAJ2GzM6MbvP/b2Sq+PgiUBcRow1f6B62CUzmJyw3ntMnNfaVWY4jRjQNgcKz0Jdwaw9Tk8tCHY+/8BFol4kQkpM4WvQIQc1iHeE7EgWVeuYj6BIAcF3RWKJFy2PcDDLNe/wOVHhfKVi1okK6DNVfB6BukyV0QdMnrpPnK2F9GKrCgiiu0FUV/twoh+NeFBblLpwdBubfp/lf9FrKGmSXB7OF0qkm7B/NfYQJwGFAfVM1fCR8ngUp5nqFIS65m45w27MMjm3XtYOn5UiO4zvqnkytfIDLrEldN/HdYBLjep+3bkqfeIpOrdNtIgvlgNgaHyeTmhts2btGLv5uIL5qNufwyGhOga7pzU4AxRvEWLUBucZZCY9Qv7ZBqMF6eXov4U3MMfPYifRaOI0DAOQHVstffXaB/5l3PTjlLRX3blGPUpxWnqIs3JXaTdCl0FQ989C0DAaHtvmkOxq0pGxdwcWgLLeNzaD1rP57bjKT/jHF1PthNe99JsjRbzcQVcrKlzgKnVUNfvCkfUHoneevBgC2TOlcS7j0qdeB2FOHOtKaE9pXr/q7M4xDaPk3n7HHev1gUaQRo7HLMItUSLmPNO6R0tlQSKN1RTg+e9qz3oIz0D4mwqmf0t7ihUttSGNsy9ARpa0kw== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: /BnprgpQhObabA1msWztPZYPGkerEwXrklU15tvhOVfA1lsyJ7+NZ//l04xGke6iAhjNpf0/mFGVhaHd3JFO7LUceuioPv4/YCtSYWBvfTkuX8C2NsLS639TXct+ocd8Q0qgGQ68v9jLKVq3ux9cqucUlLAnoIodFN7Gv2fIGjkE2ZnNRzp2mP6VfFWcQAEaBkdottdLphxlajeTYFYqCi/6iQjS1Ju1nMcgQYB3UvOhSGAMpNJPCpfh4zfqPuDv7L1B7JID+7yUy6AqXGIlExX+JN5CS4p43j3JNiJTVXyS06ETcif1MM37h90puU9YguiKbjC8vvbSoCZqA4vZ7VWJpkjAEyWMNCa5ZaL9MNJTelir/Z1TrltsT6BIS0OERkzqWTomOS4w4r0wYSOtET9IRltv7fOHvk1cG5F6I2Zouf7bfdtcv5Sgs1ln0SmpVC5CgIHf6x23ht4pLlUKADftn6WmBumvDjiqqTBjJEKw/H516uz78PQknSEDWvHd5BiKNndjCnBA6R7Y5t4IV/m3cjMr9V6l5JfmpibMfyK+X3p+Og8y7CiY3ANdt9l3cFAKlRBBQk1I+nbMylfLijuy7k4U60obilSy/zp+nDk= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0a6e25a1-1a35-4a0e-05e1-08dd5c9a71be X-MS-Exchange-CrossTenant-AuthSource: MN2PR10MB4112.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Mar 2025 10:34:16.1270 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: THWMb7JInBC+wIb8LevpBXzoeQc4O/Cn70F4pMq/dkwS8TeI3tTH3zWUUQ8qR+7VkiLS2ju0BFOTIlZ98giljLQDMMcCTV3qOLVMjV8HjDc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB4647 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1093,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-03-06_05,2025-03-06_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0 mlxscore=0 spamscore=0 bulkscore=0 suspectscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502100000 definitions=main-2503060079 X-Proofpoint-ORIG-GUID: ONm3nyZkUuZLhLNB7Jkdx_J8NlNYfuqo X-Proofpoint-GUID: ONm3nyZkUuZLhLNB7Jkdx_J8NlNYfuqo X-Rspam-User: X-Stat-Signature: kcaj35rpb86psi7ig7ypap98p96stdcq X-Rspamd-Queue-Id: 8F019120009 X-Rspamd-Server: rspam07 X-HE-Tag: 1741257260-743600 X-HE-Meta: U2FsdGVkX18nWt1b6qsYveI5TzPfbimZKkh1KEwG2ISRYK12NtE7PzSyztmJFmkrPAfdmJ2q+pewq1jaUkIxp4m3BNraYW9cwiD+g+x83EMAduTSoi5v/YwFzKVhPmhvNFjrDvNlCllCpBaSbu9B5gdyt2nrZ0dQR1JwAKK7P2vMssgpPhlWz1LLI15htI6bwvoMh1O2K+Csyft/ytPVcPAJ7U4dFxIoQehkXYPS4gRXzn11Of5XX+1zg5P4E8dM+86Y3VrPUr5ROKh6odmuNkFqRT8aPAZCfy5EIMdJX+7qox7opOJogN4oPjOqIt8BtDWLw6a2llW+vuXK8YrYd7Hq5Bupnbwa6lVMcZkHctzSpReoLsbF+kAreexqjb7HH8bRsSRcXuTsSrxNQ8HilVm/9R8+yQzIosm04jl1ZG1c6Wrpd+bsQ3asKaBlMGY4LaAeeOgWjBuYDJleIsyyCy9vcPV5ZdIlw53ClgpGHf5EnXX2XlHrAKfD7VPZRmcIjDQIn3QyabSNK6ZU/7g9xvvVBQDTqsUDhyRIJe9ragMMZxIapXJM7o2JAlBiguryAZ6P1QEDR65ioTS9Mt+6IknmP6J1WY43+D84gWHijdppIowbzQu6iuynM0OPy0VDyabwHuFntS6c+0ZYPxOU2hOXWulz+m1HLjkFj0N68tx623lCPR4MNzxjLCnUaSv9TT6e3Ze7XFIW9cRKQfCwqiKpEOTzk69gyf/jwtzyRFNPbwsgr7go6Q8Y8Qcq56eLgTZLvHrV60ZxDS9y5szFJlzNVX3O21T6qYNx2GvU4qTaZJcK1jzIn9cQd5GupoTMlibE4guGm6CL+HoL+HpBJSHciMsBZgHdRTuT8g1JUv1B5c0oWuBUO4zJw+sZBWV0xVLIhdPciCRPeThUbD++uddouMRE8S82ubLODsNkWMshmQq2C0Ff762MnoMkMTgC5Dt+FDM2bcOdCjkcuh5 1oJJHlXs X8G/PaiITKaWGvPT8sECwc4jUUsdjRCE/9sq3fwNMQBMh676zdRizH8QDgrtGciPdsPtIrCEFtDtZP/RNI2Wfeoe4+iJ6Q+zOUM7p0/aKqv871S5LUS+bXTeaGwRcPwwMt4NCwV++5fZhiC3z5KUdJow0GIU6V9X9P66S2YcHLCr5G5VfzxlBIKZePutn+QLg7DBy0AUQeqJX5iWNBhjEsqmN2h2zuwnux1hB16Rht3nAuWX65ykroow4Md1gWcPo18UrTxx6jg7v4IMu2N5OsC8OfeKRowPLEwDvCELmf/Xwk98FLhWX6lrCm/IVl8zch0HfMEuAM0IvUNVW9H29SJC1AGFhiK5pLXB8+C4l/t/kFx5bRha64fVVKKUODkDpnU7UrnRgPVnnivvWPavaCBBZyXbmcu8qoonsMnAznvbWkZISisMTwvhFbNjP+AFDnf0jBi0GKmvoKesFmZsT3Mp0P/8RmNb6myuQ2t/ZDFCn7QkAI/7TzLdWm3Jzi/1nvpR4RQx4t5iWkrwNvUuqNS07E8STDRl/nzzkeExz5WpOJh4x6PS4N6kKLKpxFyO8WHjPnUpiZ5yJQ61vZtPqXTsdB+0eIlEvhf2W4rhkO0H8QoRkxVAtBsv54I7JzB+n+w/tds6qpi8arRMfoAoVOfo3qkqCwPnQGbwZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Update move_vma() to use the threaded VRM object, de-duplicate code and separate into smaller functions to aid readability and debug-ability. This in turn allows further simplification of expand_vma() as we can simply thread VRM through the function. We also take the opportunity to abstract the account charging page count into the VRM in order that we can correctly thread this through the operation. We additionally do the same for tracking mm statistics - exec_vm, stack_vm, data_vm, and locked_vm. As part of this change, we slightly modify when locked pages statistics are counted for in mm_struct statistics. However this should cause no issues, as there is no chance of underflow, nor will any rlimit failures occur as a result. This is an intermediate step before a further refactoring of move_vma() in order to aid review. Signed-off-by: Lorenzo Stoakes Reviewed-by: Liam R. Howlett Reviewed-by: Vlastimil Babka --- mm/mremap.c | 186 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 122 insertions(+), 64 deletions(-) -- 2.48.1 diff --git a/mm/mremap.c b/mm/mremap.c index 0ab0c88072a0..9f21b468d0dc 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -68,6 +68,7 @@ struct vma_remap_struct { bool mlocked; /* Was the VMA mlock()'d? */ enum mremap_type remap_type; /* expand, shrink, etc. */ bool mmap_locked; /* Is mm currently write-locked? */ + unsigned long charged; /* If VM_ACCOUNT, # pages to account. */ }; static pud_t *get_old_pud(struct mm_struct *mm, unsigned long addr) @@ -816,35 +817,88 @@ static unsigned long vrm_set_new_addr(struct vma_remap_struct *vrm) return 0; } -static unsigned long move_vma(struct vm_area_struct *vma, - unsigned long old_addr, unsigned long old_len, - unsigned long new_len, unsigned long new_addr, - bool *mlocked, unsigned long flags, - struct vm_userfaultfd_ctx *uf, struct list_head *uf_unmap) +/* + * Keep track of pages which have been added to the memory mapping. If the VMA + * is accounted, also check to see if there is sufficient memory. + * + * Returns true on success, false if insufficient memory to charge. + */ +static bool vrm_charge(struct vma_remap_struct *vrm) { - long to_account = new_len - old_len; - struct mm_struct *mm = vma->vm_mm; - struct vm_area_struct *new_vma; - unsigned long vm_flags = vma->vm_flags; - unsigned long new_pgoff; - unsigned long moved_len; - bool account_start = false; - bool account_end = false; - unsigned long hiwater_vm; - int err = 0; - bool need_rmap_locks; - struct vma_iterator vmi; + unsigned long charged; + + if (!(vrm->vma->vm_flags & VM_ACCOUNT)) + return true; + + /* + * If we don't unmap the old mapping, then we account the entirety of + * the length of the new one. Otherwise it's just the delta in size. + */ + if (vrm->flags & MREMAP_DONTUNMAP) + charged = vrm->new_len >> PAGE_SHIFT; + else + charged = vrm->delta >> PAGE_SHIFT; + + + /* This accounts 'charged' pages of memory. */ + if (security_vm_enough_memory_mm(current->mm, charged)) + return false; + + vrm->charged = charged; + return true; +} + +/* + * an error has occurred so we will not be using vrm->charged memory. Unaccount + * this memory if the VMA is accounted. + */ +static void vrm_uncharge(struct vma_remap_struct *vrm) +{ + if (!(vrm->vma->vm_flags & VM_ACCOUNT)) + return; + + vm_unacct_memory(vrm->charged); + vrm->charged = 0; +} + +/* + * Update mm exec_vm, stack_vm, data_vm, and locked_vm fields as needed to + * account for 'bytes' memory used, and if locked, indicate this in the VRM so + * we can handle this correctly later. + */ +static void vrm_stat_account(struct vma_remap_struct *vrm, + unsigned long bytes) +{ + unsigned long pages = bytes >> PAGE_SHIFT; + struct mm_struct *mm = current->mm; + struct vm_area_struct *vma = vrm->vma; + + vm_stat_account(mm, vma->vm_flags, pages); + if (vma->vm_flags & VM_LOCKED) { + mm->locked_vm += pages; + vrm->mlocked = true; + } +} + +/* + * Perform checks before attempting to write a VMA prior to it being + * moved. + */ +static unsigned long prep_move_vma(struct vma_remap_struct *vrm, + unsigned long *vm_flags_ptr) +{ + unsigned long err = 0; + struct vm_area_struct *vma = vrm->vma; + unsigned long old_addr = vrm->addr; + unsigned long old_len = vrm->old_len; /* * We'd prefer to avoid failure later on in do_munmap: * which may split one vma into three before unmapping. */ - if (mm->map_count >= sysctl_max_map_count - 3) + if (current->mm->map_count >= sysctl_max_map_count - 3) return -ENOMEM; - if (unlikely(flags & MREMAP_DONTUNMAP)) - to_account = new_len; - if (vma->vm_ops && vma->vm_ops->may_split) { if (vma->vm_start != old_addr) err = vma->vm_ops->may_split(vma, old_addr); @@ -862,22 +916,46 @@ static unsigned long move_vma(struct vm_area_struct *vma, * so KSM can come around to merge on vma and new_vma afterwards. */ err = ksm_madvise(vma, old_addr, old_addr + old_len, - MADV_UNMERGEABLE, &vm_flags); + MADV_UNMERGEABLE, vm_flags_ptr); if (err) return err; - if (vm_flags & VM_ACCOUNT) { - if (security_vm_enough_memory_mm(mm, to_account >> PAGE_SHIFT)) - return -ENOMEM; - } + return 0; +} + +static unsigned long move_vma(struct vma_remap_struct *vrm) +{ + struct mm_struct *mm = current->mm; + struct vm_area_struct *vma = vrm->vma; + struct vm_area_struct *new_vma; + unsigned long vm_flags = vma->vm_flags; + unsigned long old_addr = vrm->addr, new_addr = vrm->new_addr; + unsigned long old_len = vrm->old_len, new_len = vrm->new_len; + unsigned long new_pgoff; + unsigned long moved_len; + unsigned long account_start = false; + unsigned long account_end = false; + unsigned long hiwater_vm; + int err; + bool need_rmap_locks; + struct vma_iterator vmi; + + err = prep_move_vma(vrm, &vm_flags); + if (err) + return err; + + /* If accounted, charge the number of bytes the operation will use. */ + if (!vrm_charge(vrm)) + return -ENOMEM; vma_start_write(vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT); - new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff, + new_vma = copy_vma(&vrm->vma, new_addr, new_len, new_pgoff, &need_rmap_locks); + /* This may have been updated. */ + vma = vrm->vma; if (!new_vma) { - if (vm_flags & VM_ACCOUNT) - vm_unacct_memory(to_account >> PAGE_SHIFT); + vrm_uncharge(vrm); return -ENOMEM; } @@ -902,7 +980,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, old_addr = new_addr; new_addr = err; } else { - mremap_userfaultfd_prep(new_vma, uf); + mremap_userfaultfd_prep(new_vma, vrm->uf); } if (is_vm_hugetlb_page(vma)) { @@ -910,7 +988,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, } /* Conceal VM_ACCOUNT so old reservation is not undone */ - if (vm_flags & VM_ACCOUNT && !(flags & MREMAP_DONTUNMAP)) { + if (vm_flags & VM_ACCOUNT && !(vrm->flags & MREMAP_DONTUNMAP)) { vm_flags_clear(vma, VM_ACCOUNT); if (vma->vm_start < old_addr) account_start = true; @@ -928,13 +1006,12 @@ static unsigned long move_vma(struct vm_area_struct *vma, * If this were a serious issue, we'd add a flag to do_munmap(). */ hiwater_vm = mm->hiwater_vm; - vm_stat_account(mm, vma->vm_flags, new_len >> PAGE_SHIFT); /* Tell pfnmap has moved from this vma */ if (unlikely(vma->vm_flags & VM_PFNMAP)) untrack_pfn_clear(vma); - if (unlikely(!err && (flags & MREMAP_DONTUNMAP))) { + if (unlikely(!err && (vrm->flags & MREMAP_DONTUNMAP))) { /* We always clear VM_LOCKED[ONFAULT] on the old vma */ vm_flags_clear(vma, VM_LOCKED_MASK); @@ -947,22 +1024,20 @@ static unsigned long move_vma(struct vm_area_struct *vma, unlink_anon_vmas(vma); /* Because we won't unmap we don't need to touch locked_vm */ + vrm_stat_account(vrm, new_len); return new_addr; } + vrm_stat_account(vrm, new_len); + vma_iter_init(&vmi, mm, old_addr); - if (do_vmi_munmap(&vmi, mm, old_addr, old_len, uf_unmap, false) < 0) { + if (do_vmi_munmap(&vmi, mm, old_addr, old_len, vrm->uf_unmap, false) < 0) { /* OOM: unable to split vma, just get accounts right */ - if (vm_flags & VM_ACCOUNT && !(flags & MREMAP_DONTUNMAP)) + if (vm_flags & VM_ACCOUNT && !(vrm->flags & MREMAP_DONTUNMAP)) vm_acct_memory(old_len >> PAGE_SHIFT); account_start = account_end = false; } - if (vm_flags & VM_LOCKED) { - mm->locked_vm += new_len >> PAGE_SHIFT; - *mlocked = true; - } - mm->hiwater_vm = hiwater_vm; /* Restore VM_ACCOUNT if one or two pieces of vma left */ @@ -1136,9 +1211,7 @@ static unsigned long mremap_to(struct vma_remap_struct *vrm) if (err) return err; - return move_vma(vrm->vma, vrm->addr, vrm->old_len, vrm->new_len, - vrm->new_addr, &vrm->mlocked, vrm->flags, - vrm->uf, vrm->uf_unmap); + return move_vma(vrm); } static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) @@ -1243,17 +1316,11 @@ static unsigned long check_mremap_params(struct vma_remap_struct *vrm) static unsigned long expand_vma_in_place(struct vma_remap_struct *vrm) { struct mm_struct *mm = current->mm; - long pages = vrm->delta >> PAGE_SHIFT; struct vm_area_struct *vma = vrm->vma; VMA_ITERATOR(vmi, mm, vma->vm_end); - long charged = 0; - if (vma->vm_flags & VM_ACCOUNT) { - if (security_vm_enough_memory_mm(mm, pages)) - return -ENOMEM; - - charged = pages; - } + if (!vrm_charge(vrm)) + return -ENOMEM; /* * Function vma_merge_extend() is called on the @@ -1266,15 +1333,11 @@ static unsigned long expand_vma_in_place(struct vma_remap_struct *vrm) */ vma = vrm->vma = vma_merge_extend(&vmi, vma, vrm->delta); if (!vma) { - vm_unacct_memory(charged); + vrm_uncharge(vrm); return -ENOMEM; } - vm_stat_account(mm, vma->vm_flags, pages); - if (vma->vm_flags & VM_LOCKED) { - mm->locked_vm += pages; - vrm->mlocked = true; - } + vrm_stat_account(vrm, vrm->delta); return 0; } @@ -1314,11 +1377,7 @@ static bool align_hugetlb(struct vma_remap_struct *vrm) static unsigned long expand_vma(struct vma_remap_struct *vrm) { unsigned long err; - struct vm_area_struct *vma = vrm->vma; unsigned long addr = vrm->addr; - unsigned long old_len = vrm->old_len; - unsigned long new_len = vrm->new_len; - unsigned long flags = vrm->flags; err = resize_is_valid(vrm); if (err) @@ -1351,7 +1410,7 @@ static unsigned long expand_vma(struct vma_remap_struct *vrm) */ /* We're not allowed to move the VMA, so error out. */ - if (!(flags & MREMAP_MAYMOVE)) + if (!(vrm->flags & MREMAP_MAYMOVE)) return -ENOMEM; /* Find a new location to move the VMA to. */ @@ -1359,8 +1418,7 @@ static unsigned long expand_vma(struct vma_remap_struct *vrm) if (err) return err; - return move_vma(vma, addr, old_len, new_len, vrm->new_addr, - &vrm->mlocked, flags, vrm->uf, vrm->uf_unmap); + return move_vma(vrm); } /* From patchwork Thu Mar 6 10:34:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Stoakes X-Patchwork-Id: 14004153 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A0B3C282D1 for ; Thu, 6 Mar 2025 10:34:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 93034280007; Thu, 6 Mar 2025 05:34:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DEB8280003; Thu, 6 Mar 2025 05:34:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5668F280008; Thu, 6 Mar 2025 05:34:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2790D280007 for ; Thu, 6 Mar 2025 05:34:28 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C406716140A for ; Thu, 6 Mar 2025 10:34:28 +0000 (UTC) X-FDA: 83190767016.16.88DF104 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf06.hostedemail.com (Postfix) with ESMTP id 54666180003 for ; Thu, 6 Mar 2025 10:34:25 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=mrU+Bf3A; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=OcoAzVVg; spf=pass (imf06.hostedemail.com: domain of lorenzo.stoakes@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=lorenzo.stoakes@oracle.com; dmarc=pass (policy=reject) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741257265; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8KOwdcZdySxF73nLwM+suOaK5y/3OXix4HdEU9hYnXk=; b=jLpKPqIixLbjiOTQJvHPm96CLzwKSe52Gx6LF+88su2lhNL9SKzADwbI/9aKh9T0EZ2bzh WZboaIQN+cwR92dBOx+lc3bQr+XFOlB5ypbnU8tBXQhRHNTcPoRCpwvena5e2lrGnn6dDu 4J5yMiJmMdPRsDcbKis8sm5ta7LtBvA= ARC-Authentication-Results: i=2; imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=mrU+Bf3A; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=OcoAzVVg; spf=pass (imf06.hostedemail.com: domain of lorenzo.stoakes@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=lorenzo.stoakes@oracle.com; dmarc=pass (policy=reject) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1741257265; a=rsa-sha256; cv=pass; b=8F4YxbqXnNsj7Hc5QD9PY+DdcLYbNBkH3it8ZanFmx4af0zG4CG9eSp1VKVzCOQv/tx131 Sa6u1ufIV1xCP60pMaMmQNnBXcNRIUQYzDpWXu58JJE3V5Pw9KrEBtJhePGUolfAXvZQkN Rsp/qrJTIuc52efnvkwVixDKfRpG9n0= Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 526ABlfS000825; Thu, 6 Mar 2025 10:34:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=8KOwdcZdySxF73nLwM+suOaK5y/3OXix4HdEU9hYnXk=; b= mrU+Bf3A1I5RZ0SKpOd3dcccXAVMeT5NkCqbADHroTQu4FbcxoCKLaFbih2s7vzU Wq7ukb42g4oe8kq7oOBt/IFupBL7o7sN55XtBa8er4p4T+pARRUNFqL7F24Fz8Al ENURFLv1blVi9iFgVXbEh+UrO6919WIB6sZnG3VOZkC3/praz4c8ErHaUi8/eI0E Tq5k7IFl/oQBUAoFvQQ8U1XfvT0UCSdEQzTspRsjYNoXEUqPJ7WbCZtiS0yFb8fy XS9GOsezo8rG8QPTKkvbKt/zk5JsjK+3tEzinf9NHrPzXsOnQG3TRD/1OtdWoEu2 TM+xW0zQkZMmLaxZ4G/0gQ== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 453u821v6t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 06 Mar 2025 10:34:21 +0000 (GMT) Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 526AJ5N5010984; Thu, 6 Mar 2025 10:34:20 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2171.outbound.protection.outlook.com [104.47.57.171]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 453rpdh6t6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 06 Mar 2025 10:34:20 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=r1bjbba1F5zqVcQb+JdgCAl+EQMOw+UZGpbCQ0fk7Wd6pfOTJvpZ8eTsMNbboL/VLjV+kfwRUFRDFBPJP4Hkvxsvh3ZXyqmZhv6Z48A2mcuH7v7tfViG7y+AzBBXttEzNoBs5roNiorHWleGbUKrhpsMZoAfAXv+ajJ7Qm/95gfAgu35sXPGsIVzQNDD+dNsCeFSyTIlYHSb2xZsF9g8HAuiy+sBicU5O9LmHbyyr0ijG7s6iLean/i2weIt8YnGBIuRLTUyyKw+tnJhMvttYEsJO+F+SObTnfqDUkIy1fHsXZFFkMPHdh6PGOmQBWw1IMnwGls5EXANN9AEfPKM2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8KOwdcZdySxF73nLwM+suOaK5y/3OXix4HdEU9hYnXk=; b=p9WwXbvCRaWE3I44iSKv9Ydmrxrot/z5VfuXD/7CuLXDEJ8kXTDjSvJfx22ZYHyCbXKmKvNXXJzwsEmUHSbiqeOZsB+ghh0NlqTj9c8dGUdeNjJIritJxhzuDRXLFNLhfayn6u2VfiW6q+duJdw1T7f+FeyKz41RIanBxr5MbUuo4FPtxfxUSpsm0zZJGR+BdHqX4GsYMYjlruLIFWXf+ita/yL/6x7LPFma36IcIuPOn+nj8AIQRGg8m0pfDyxfx/Xfi9ZGSxbfAPu1AQjEM5r+DXYropPxtTRdWRl0HW5NVXlSxn2qv9D8Bh15LVuVFmzGgXQXt10KdwcvL/FyUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8KOwdcZdySxF73nLwM+suOaK5y/3OXix4HdEU9hYnXk=; b=OcoAzVVgRlrwC11xbURMSqxmVHmKGJ//nEWdI2Z414EIoaC+qjPTLfd1jYDuAvKDmSfPkjj4vZd76atCFFgpLowvmBNOPIwrxQpw/Ecjs9WzenRlrPg2wfN8WFcBU/PvnrNqbp87z52j2f4l8mpbvr4ptDvudH2diERQ5sJEvb4= Received: from MN2PR10MB4112.namprd10.prod.outlook.com (2603:10b6:208:11e::33) by PH0PR10MB4647.namprd10.prod.outlook.com (2603:10b6:510:43::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.20; Thu, 6 Mar 2025 10:34:18 +0000 Received: from MN2PR10MB4112.namprd10.prod.outlook.com ([fe80::3256:3c8c:73a9:5b9c]) by MN2PR10MB4112.namprd10.prod.outlook.com ([fe80::3256:3c8c:73a9:5b9c%7]) with mapi id 15.20.8489.025; Thu, 6 Mar 2025 10:34:18 +0000 From: Lorenzo Stoakes To: Andrew Morton Cc: "Liam R . Howlett" , Vlastimil Babka , Jann Horn , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Harry Yoo , Yosry Ahmed Subject: [PATCH v2 5/7] mm/mremap: complete refactor of move_vma() Date: Thu, 6 Mar 2025 10:34:01 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: X-ClientProxiedBy: LO4P123CA0678.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:351::18) To MN2PR10MB4112.namprd10.prod.outlook.com (2603:10b6:208:11e::33) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN2PR10MB4112:EE_|PH0PR10MB4647:EE_ X-MS-Office365-Filtering-Correlation-Id: 04162979-2f0b-4e4f-3284-08dd5c9a72fc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|366016; X-Microsoft-Antispam-Message-Info: pKDvZL+RB0Nyhq0ZwAmQ3QI7lLq+tcrSgraDerjAtbcCiEJwxaQ2kvi5KBLc4bFz4BqCC85PzH9CBXp9hPsSqz2eRv3o6YpDxt376vUQ4aqHJhVZLZBFWdYNULV/YyEp5BIHW2XOxvmpAsxTqZkxA345fV8xKtQ/B9OYY/BY6XHa/9dyyrFCEBBRKVp3tzkjj4cs2lropCv5Cbk4oLZokNcP/N2fWimA9f+7RtliPcsDSI+9Uz4PaopV1vjun7C8W+njASBkCFaWJs+OxI4kHvZYuC+ZPhQw/dPHse+apq7DSWAVspR/Wvi8OQ/f+EgPQEbAiW6UsVS+bOds2uIhSX9H3kwgt8S4ssLVMHkMahiHbQpEPuCrpCjR4PdNQNj3nW4O0afUMlOaKMlakS9Wqk+LP0P5iAwqetcbjHVFQha0NW0WWW4BHDgsvaI1Lm7RffZoWqxXiEXTfG1WdPrElA7OEEGEBPeNsOOeGYko2U7+CN15VcMQGM40cdcaA70JqdyK2bGtQUcKSzn1h1iXwhVMg68Wkcn2h0Z57YgVM2Og6MhbmRECbxH6fGGy1Lsmc5lpdrRlw7+NfeIpNSkZ+FYs8VTBIoq7iu0txSS5MRglZAy0LM3OcQlZbNEi/sBDFfvMl4ow/PQgd/YDmB3vXGdfl6eah+lY1W33DHhuLAMnWlQLRzGNds81msmMiL2zDX1ARlxApV4Gsq0b+RIHZlkdb96/Bfk5ngr7IGkU9cBFR0NocsXnzjOqxv6vzPhqF4cwr703oPps5ALY/k8J+PoF3PWlBY15g9S8K2uBK9EgQyOTG7Bu4Oht6a0/qDar5SeGat6f67baErT3uBOWqVL/hGHS9rh540CVxSVRXLrVgYIczeS6ktwyxhPLo1OBhbhy1aXghmuQ9UR+kLJLFxwzFZXZW0fp2Aa3RWA3U1wV63nTqsGD/qXI4AhFrzvvp3kgj8HgHSnarB++2wyMXweEh5BdPlK8DHtRD4HuVzcG6Je/jxLvj4ZkgL8NhMf0Qz5IuhcY/X9U0gI7zlOlw6oc0tBSvB9y8q2rNqZhfJlJTra2NnlX2ETfqDZpRpe9b7fzoktZnCUcttf7aUtPXVdTbpVCS/i8V4ROQhMxYmy/0Nd/41lTEzlqJB1O3NQ0I0k3qegZZv2if4zknTzoflGW67IkTAY3tQMtWfy0omsVpsLvn2O/kqvngsbRfgg2TuB901UubfE61oOLTXqgeze+2Ias+KP9Rk2A2+Nze0VABVn/3XHWX0cU+flY/lB98ifVuZot5z440Vw7k6Z9m3yyqHJv913JXIY4enM0o34a5nCN0fLZdYp3+Kjfh1g0aLghvNDnxI+Lotl0b9+yBATinW6zsQVGgn+KHbCgVdVxQ8cfczguOhmzLYCuLnU9 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MN2PR10MB4112.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: gX7NpOgc3DbAokVBuDDGJJ6vv089fZRjtbMo3Ba7TZF9biNNEJw8zRMbOmlhlRCVVsNfLoL6z7nJ0cagqXYZhkWGJ4WWOkWfFM5/Qt22C5cO6WMGI/Ji+ECyXObmQsQkudmcryLLUak6uyamILyKovw5pDfpG8xTdEIKf0T6PqKTPq5QTFcv1oZE0oy9b9v/G4ZLmy2lIaqB3f9Ph06WupknPK/pSeiAQ7ZBv9lk0Zg7SpyonHHa8AR6U011XWILI1yo3MI5gq5I4dIO5QbZQ6cURQnibFjv6k9X5Okm5v6B3NUpedU+rJkBQIG+2GilRmuzKK/e1hxx+X7UBeiGGGuWMmZp4U+gqO+ULnFVedR0zW9Uu6zbATDOaWoL4Wvx2W7D80dSHQm1O+8GbO6xwdJYpciUAzwx4/mDaru+1dAfK6P4rni4nt6GlE4/xEuUi6O67rBcsHx1Lkaczmn7kFtmU/7e0c4/byF8kAm+2oErMZuYlRdSSN7cgxW8n39G4ONxEjTezP+MfFfVBcb2zyP9Cybq+39QR8zWAWNnbfdY1WVEoilpZTgvG7blw6ZwFoIOALcBPpwafQ6ujdjYlWVh9lgpEmsYfGJ3JkE5c0OhbDOKG87x5OvRRaBiO3SSS0gm1vEy26ChsLnPXjTdLgvWWFz1SF5l4XT5YaIwzE9Wk5Xm2PxMQtdvmv08UD6ADDEnTyRF6w1vQXFDg4OsVWNhIVx/e5ZPtfTmEn8O44CvHFHqNoOzO5HTTsUNNddKU3aWKCpNI65A8Z72xDy0QpjxPJYyL67CFmdVSdDBfOcJ1/yRzn57Er5sHdZXOlZ4hNlBij1EvNVPkhjtIoN6WgewRgd6YOtXWuKlVWrhAoYTNHVkNfM/2pdIpkfFoG0ZkS1X4yG54YeVzCfI6pxV1LeaxRRQDsVc/cIImGaibFOcZDHrkbhHCNnmHLAftCq6IVPkcjeZH33ssbpuuDTVvC6iRwClgQQzsyeBSXlCza+SU/E21rzQZEGimZUTIMXgX4sjFU0iIHmXDF7w56h9vkWo+gWfNhTQP6rz4pujjkGKTHSn/pDyl67/QNYrkdb35zaGcDtEWC3vJrh8GN+QEvHaLnP0nNAEAlSRj8JGzmuu8EeSd49pyToKiA2CBD6L7NRaF2VD2I+0VpiDzEInrmd2F5nUPb2lrlGh3BxT6Oh3XJg3Y9dB1VRPe0ImnqUPZF3btWbzwQBsatnzlj4fNbd0Uu6f0G0h/1OkktP4wfOO+2hHgfNBLr4PsYAmGbfxy4aL3UOh7Z6mHTrY5OZDPHzo+ocDPbPbmVvbcl0lSe410XmovQ/1qiGxVYoQWrQivSddZMFnENSpGYPqTCImHwWUjTzmA0qzgrQ6V0gwi913cXg0SAFwE/6lokDq7VHnWUdcWU4xS8MtmBf0qRQy65mCBZw4qzmaqlrt7XqxhhLr8k2qdIXZQCnUkVG+vY0n3k3vWl6a7FW2nvp9RMfnqfH/FC6RKLCA+SVZJXNPC6Cx7Y4M9nD6oxxX3cjJsjpnoH1dZclDxQ1zVY/j6hJRmkmq+AcPGEXjOVCJo6HDFqPnjB81jDKSjgN7+LZvV/t4MZh+mUrEmv8XZKZ9ufViOQ== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 0yNigm3Zt1ibHkgTNqHu2/w9gc6E208y5BZtR2xEXtDyVC4Xnr6S3xzgwpTeoJ4zMRj0UjSm9gTziuMClvNrbnIDYsy1YIeqOgzYYpbUPGK6NO51/TqHK/R2o4L5JaFGBNqK+q6e8lunHbzcVZQyzhbwJjCmkrLZkhGt0tEh0bWFKm/cQ0jmwAKRa0Fz1ldTqKm45N8HZgRAYZN3SX1rMo6YKqAqxBrapjctR/PzRJgcJCDZsxh21B6Lc4XYTWp2w8giIY+cnKO35Kfuo2HnUO0ZzcEeg/rq7PjPS1y/gEQy8Ggp013laZSmONFEjzExwxIhyrRnzp7wf0OiQi1jL8K/ox8zlnX/URosgqkIuaKq+etSwb7kA4n/wkP5hBw1+Iwf6q88EiW4jZHiPN+RbMzN8iKGfqEmMhllkeEFkXunfeqb7UHzDcOoRcAlmaTfsO11nC4KWMRCHDWPGbwJ2nvR1mk1mfsb9efcTrPYhv6a8QvPpnbg6LSNna1QmcXkdhPfF7RaWTdrRak4QbJkXm/Q9pgLyg2upOlW4Hrk4TxOAdEWbkwvwep/5ZtEFsvXArj2oVxAYBfJinuKZIdSeOQsFI9mypTB/UF6J53DL90= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 04162979-2f0b-4e4f-3284-08dd5c9a72fc X-MS-Exchange-CrossTenant-AuthSource: MN2PR10MB4112.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Mar 2025 10:34:18.2144 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cjSP0qlscp8YHaMA/dGg/YSowLm3GthSE1L7Mg6MMmiDo2zWdbsogox1/2nQQ02JOvWPrxEEEC9r6bKaOrydlCX7LKkxs+ujgWxDMCYju/c= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB4647 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1093,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-03-06_05,2025-03-06_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0 mlxscore=0 spamscore=0 bulkscore=0 suspectscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502100000 definitions=main-2503060079 X-Proofpoint-GUID: jjy4k6qFOEU3A8m5H3QfKZtw1lrPuIxj X-Proofpoint-ORIG-GUID: jjy4k6qFOEU3A8m5H3QfKZtw1lrPuIxj X-Rspam-User: X-Stat-Signature: 9xnzxh6fkg4nowege1jcm3ujn6h9cf8t X-Rspamd-Queue-Id: 54666180003 X-Rspamd-Server: rspam07 X-HE-Tag: 1741257265-2196 X-HE-Meta: U2FsdGVkX1/73TpQTnv5wu/YNwnP66FLIVZKBjJkYbp+srGxGm6EF8ttGPPG/sGpDcZDIHG9AkJZ1t0B0ljTka0+SsJnikT+JFHobUHowtDNODS3/Ls8BCgZy2tOqpRSZYXnQMRGMLOSAZav54xDBZMSl0bcLHgvpSt73zgODqrdvNBrw6D7mTvMQHkjmyiWcPKnSeevGmPNshuBdhiJ8S8j3X147Ozc0rgRvBAyk2rY765ITtmWtu+aeKlof/fTRyzqY5m9bfm+f4ADBHRzx8oZrBZo7Sx/FPiWGpbc6/cyaxh/g/G8cW/JnvWz3ldm9av3Jr/AJTXMlvFUVq9U7qNTkJI3p+SrOO02lwreaqqVOISuXs8by8dNa98VoHPxizYGi6sNtUliZ7kxgKLN/bAVhkvi9kSeNuaVYB9ePnkbBrCJsK01GBfhESP3VH3ORtCeXt8LJhv9+ylvYnDsmYqCu316YK4F6Cn8QFs77hUZ+5npJ/P5z943pywVOGE1kNSVtqsaIHdMGSqDONpuWKhivVqqbxKSDHJSYlbjRrMdJSLMIX1UuTl1KjsJuhH8E4knJl8eCZs0p5t2bdhD4tELHCGY4w4kHkL1kPjdDwcaljT314GoKeCECeykrblYVdv9tJ7uKcCI28c+1ZQAQ+vx/GLkeHXou1g4x/7zA/nAoIBF7LBtyY5uwLIUspjB8I9kROemdpF9n86AWY0yHP/WXTtBqxD/caUzhJIFjzylMCe5f5UKd8xzO4+y7xyMwF8yMVCojZw2D8IZi1onIMC+3YKGzQbKu7VoLomp+Kwgq6eaX+r9lsXTL4ALIqnx01kIhTqCPgpe7IkjGaonajrMxDODk7II+SX+IV6bztjddO3TT3J7z8ObYD1gIeqrO5Fa/6VO6s8umOZ38wo+ZYSBPrIOlYQf547Q+R9A7uYw/VDOeqfuC8tlsxD/MIauXZcdwpsz+emR9NsMW3U PNUdiR4N Ck9x5fP5gNi1Jo93oWruFy9HX/hA/tyK4PI4RWKn+KAaB8GsOmFB0iFrckfP5SVftMhNNvPMATqW6dDl//JMTAJENTYWilfzHSN6Ar90y63Px5YAgHYZ1FkvxQfwao0xBp1XbjvjUZBWeE8PsapL/o8uBuXrGEhlx+WUro0C4K3yDpqfPoC+c9FcJm2AA+P8kHTD93Q6Yagkh/ouu/U8Y/hqdkBrnN2swvvvA2xDdI3coeo+SL4D0najsCH28hsYTAp5iiksJfRbPWbyRN6YXkHIsJ3eei8sx4l+cD/MVaSt8CN+9wYt5CtyWqOxP0zn/deuLNs9Fh4cJ1dAwR0zRyhKKWDZRDNOMeCIYZJDSOtG01GJBlXUDcvFumKzPJ4d1WWIXtkZuIuV7qmp2KQa3oZR4ja55xLRigySuqpmZnYexXTusPzzOk30o+yZBu76G8BhHzuYJGDO0umAVCh6YyUdCTbPbhLQP2YmB+xQpXYzwl3w2/zoLKEyjazMGegUx3q9Yp/ztVhxyaKbRKbLAOrHg1hRTmLM/YA+WXCRNgmBEYGeruuyCknCDhqxLq5UMMkaNZcuJwuUawkqXN5R0fmfaR+dRzn3s6MiQhUOPd+tL6ZV411zk74SjaUHiVypZfJWEFj4shOsu9dV+JUM/VJ9s/IsVnOrtKw76 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We invoke ksm_madvise() with an intentionally dummy flags field, so no need to pass around. Additionally, the code tries to be 'clever' with account_start, account_end, using these to both check that vma->vm_start != 0 and that we ought to account the newly split portion of VMA post-move, either before or after it. We need to do this because we intentionally removed VM_ACCOUNT on the VMA prior to unmapping, so we don't erroneously unaccount memory (we have already calculated the correct amount to account and accounted it, any subsequent subtraction will be incorrect). This patch significantly expands the comment (from 2002!) about 'concealing' the flag to make it abundantly clear what's going on, as well as adding and expanding a number of other comments also. We can remove account_start, account_end by instead tracking when we account (i.e. vma->vm_flags has the VM_ACCOUNT flag set, and this is not an MREMAP_DONTUNMAP operation), and figuring out when to reinstate the VM_ACCOUNT flag on prior/subsequent VMAs separately. We additionally break the function into logical pieces and attack the very confusing error handling logic (where, for instance, new_addr is set to err). After this change the code is considerably more readable and easy to manipulate. Signed-off-by: Lorenzo Stoakes Reviewed-by: Vlastimil Babka --- mm/mremap.c | 292 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 203 insertions(+), 89 deletions(-) -- 2.48.1 diff --git a/mm/mremap.c b/mm/mremap.c index 9f21b468d0dc..df550780a450 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -884,13 +884,13 @@ static void vrm_stat_account(struct vma_remap_struct *vrm, * Perform checks before attempting to write a VMA prior to it being * moved. */ -static unsigned long prep_move_vma(struct vma_remap_struct *vrm, - unsigned long *vm_flags_ptr) +static unsigned long prep_move_vma(struct vma_remap_struct *vrm) { unsigned long err = 0; struct vm_area_struct *vma = vrm->vma; unsigned long old_addr = vrm->addr; unsigned long old_len = vrm->old_len; + unsigned long dummy = vma->vm_flags; /* * We'd prefer to avoid failure later on in do_munmap: @@ -916,56 +916,150 @@ static unsigned long prep_move_vma(struct vma_remap_struct *vrm, * so KSM can come around to merge on vma and new_vma afterwards. */ err = ksm_madvise(vma, old_addr, old_addr + old_len, - MADV_UNMERGEABLE, vm_flags_ptr); + MADV_UNMERGEABLE, &dummy); if (err) return err; return 0; } -static unsigned long move_vma(struct vma_remap_struct *vrm) +/* + * Unmap source VMA for VMA move, turning it from a copy to a move, being + * careful to ensure we do not underflow memory account while doing so if an + * accountable move. + * + * This is best effort, if we fail to unmap then we simply try + */ +static void unmap_source_vma(struct vma_remap_struct *vrm) { struct mm_struct *mm = current->mm; + unsigned long addr = vrm->addr; + unsigned long len = vrm->old_len; struct vm_area_struct *vma = vrm->vma; - struct vm_area_struct *new_vma; - unsigned long vm_flags = vma->vm_flags; - unsigned long old_addr = vrm->addr, new_addr = vrm->new_addr; - unsigned long old_len = vrm->old_len, new_len = vrm->new_len; - unsigned long new_pgoff; - unsigned long moved_len; - unsigned long account_start = false; - unsigned long account_end = false; - unsigned long hiwater_vm; + VMA_ITERATOR(vmi, mm, addr); int err; - bool need_rmap_locks; - struct vma_iterator vmi; + unsigned long vm_start; + unsigned long vm_end; + /* + * It might seem odd that we check for MREMAP_DONTUNMAP here, given this + * function implies that we unmap the original VMA, which seems + * contradictory. + * + * However, this occurs when this operation was attempted and an error + * arose, in which case we _do_ wish to unmap the _new_ VMA, which means + * we actually _do_ want it be unaccounted. + */ + bool accountable_move = (vma->vm_flags & VM_ACCOUNT) && + !(vrm->flags & MREMAP_DONTUNMAP); - err = prep_move_vma(vrm, &vm_flags); - if (err) - return err; + /* + * So we perform a trick here to prevent incorrect accounting. Any merge + * or new VMA allocation performed in copy_vma() does not adjust + * accounting, it is expected that callers handle this. + * + * And indeed we already have, accounting appropriately in the case of + * both in vrm_charge(). + * + * However, when we unmap the existing VMA (to effect the move), this + * code will, if the VMA has VM_ACCOUNT set, attempt to unaccount + * removed pages. + * + * To avoid this we temporarily clear this flag, reinstating on any + * portions of the original VMA that remain. + */ + if (accountable_move) { + vm_flags_clear(vma, VM_ACCOUNT); + /* We are about to split vma, so store the start/end. */ + vm_start = vma->vm_start; + vm_end = vma->vm_end; + } - /* If accounted, charge the number of bytes the operation will use. */ - if (!vrm_charge(vrm)) - return -ENOMEM; + err = do_vmi_munmap(&vmi, mm, addr, len, vrm->uf_unmap, /* unlock= */false); + vrm->vma = NULL; /* Invalidated. */ + if (err) { + /* OOM: unable to split vma, just get accounts right */ + vm_acct_memory(len >> PAGE_SHIFT); + return; + } - vma_start_write(vma); - new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT); - new_vma = copy_vma(&vrm->vma, new_addr, new_len, new_pgoff, + /* + * If we mremap() from a VMA like this: + * + * addr end + * | | + * v v + * |-------------| + * | | + * |-------------| + * + * Having cleared VM_ACCOUNT from the whole VMA, after we unmap above + * we'll end up with: + * + * addr end + * | | + * v v + * |---| |---| + * | A | | B | + * |---| |---| + * + * The VMI is still pointing at addr, so vma_prev() will give us A, and + * a subsequent or lone vma_next() will give as B. + * + * do_vmi_munmap() will have restored the VMI back to addr. + */ + if (accountable_move) { + unsigned long end = addr + len; + + if (vm_start < addr) { + struct vm_area_struct *prev = vma_prev(&vmi); + + vm_flags_set(prev, VM_ACCOUNT); /* Acquires VMA lock. */ + } + + if (vm_end > end) { + struct vm_area_struct *next = vma_next(&vmi); + + vm_flags_set(next, VM_ACCOUNT); /* Acquires VMA lock. */ + } + } +} + +/* + * Copy vrm->vma over to vrm->new_addr possibly adjusting size as part of the + * process. Additionally handle an error occurring on moving of page tables, + * where we reset vrm state to cause unmapping of the new VMA. + * + * Outputs the newly installed VMA to new_vma_ptr. Returns 0 on success or an + * error code. + */ +static int copy_vma_and_data(struct vma_remap_struct *vrm, + struct vm_area_struct **new_vma_ptr) +{ + unsigned long internal_offset = vrm->addr - vrm->vma->vm_start; + unsigned long internal_pgoff = internal_offset >> PAGE_SHIFT; + unsigned long new_pgoff = vrm->vma->vm_pgoff + internal_pgoff; + unsigned long moved_len; + bool need_rmap_locks; + struct vm_area_struct *vma; + struct vm_area_struct *new_vma; + int err = 0; + + new_vma = copy_vma(&vrm->vma, vrm->new_addr, vrm->new_len, new_pgoff, &need_rmap_locks); - /* This may have been updated. */ - vma = vrm->vma; if (!new_vma) { vrm_uncharge(vrm); + *new_vma_ptr = NULL; return -ENOMEM; } + vma = vrm->vma; - moved_len = move_page_tables(vma, old_addr, new_vma, new_addr, old_len, - need_rmap_locks, false); - if (moved_len < old_len) { + moved_len = move_page_tables(vma, vrm->addr, new_vma, + vrm->new_addr, vrm->old_len, + need_rmap_locks, /* for_stack= */false); + if (moved_len < vrm->old_len) err = -ENOMEM; - } else if (vma->vm_ops && vma->vm_ops->mremap) { + else if (vma->vm_ops && vma->vm_ops->mremap) err = vma->vm_ops->mremap(new_vma); - } if (unlikely(err)) { /* @@ -973,28 +1067,84 @@ static unsigned long move_vma(struct vma_remap_struct *vrm) * which will succeed since page tables still there, * and then proceed to unmap new area instead of old. */ - move_page_tables(new_vma, new_addr, vma, old_addr, moved_len, - true, false); - vma = new_vma; - old_len = new_len; - old_addr = new_addr; - new_addr = err; + move_page_tables(new_vma, vrm->new_addr, vma, vrm->addr, + moved_len, /* need_rmap_locks = */true, + /* for_stack= */false); + vrm->vma = new_vma; + vrm->old_len = vrm->new_len; + vrm->addr = vrm->new_addr; } else { mremap_userfaultfd_prep(new_vma, vrm->uf); } - if (is_vm_hugetlb_page(vma)) { + if (is_vm_hugetlb_page(vma)) clear_vma_resv_huge_pages(vma); - } - /* Conceal VM_ACCOUNT so old reservation is not undone */ - if (vm_flags & VM_ACCOUNT && !(vrm->flags & MREMAP_DONTUNMAP)) { - vm_flags_clear(vma, VM_ACCOUNT); - if (vma->vm_start < old_addr) - account_start = true; - if (vma->vm_end > old_addr + old_len) - account_end = true; - } + /* Tell pfnmap has moved from this vma */ + if (unlikely(vma->vm_flags & VM_PFNMAP)) + untrack_pfn_clear(vma); + + *new_vma_ptr = new_vma; + return err; +} + +/* + * Perform final tasks for MADV_DONTUNMAP operation, clearing mlock() and + * account flags on remaining VMA by convention (it cannot be mlock()'d any + * longer, as pages in range are no longer mapped), and removing anon_vma_chain + * links from it (if the entire VMA was copied over). + */ +static void dontunmap_complete(struct vma_remap_struct *vrm, + struct vm_area_struct *new_vma) +{ + unsigned long start = vrm->addr; + unsigned long end = vrm->addr + vrm->old_len; + unsigned long old_start = vrm->vma->vm_start; + unsigned long old_end = vrm->vma->vm_end; + + /* + * We always clear VM_LOCKED[ONFAULT] | VM_ACCOUNT on the old + * vma. + */ + vm_flags_clear(vrm->vma, VM_LOCKED_MASK | VM_ACCOUNT); + + /* + * anon_vma links of the old vma is no longer needed after its page + * table has been moved. + */ + if (new_vma != vrm->vma && start == old_start && end == old_end) + unlink_anon_vmas(vrm->vma); + + /* Because we won't unmap we don't need to touch locked_vm. */ +} + +static unsigned long move_vma(struct vma_remap_struct *vrm) +{ + struct mm_struct *mm = current->mm; + struct vm_area_struct *new_vma; + unsigned long hiwater_vm; + int err; + + err = prep_move_vma(vrm); + if (err) + return err; + + /* If accounted, charge the number of bytes the operation will use. */ + if (!vrm_charge(vrm)) + return -ENOMEM; + + /* We don't want racing faults. */ + vma_start_write(vrm->vma); + + /* Perform copy step. */ + err = copy_vma_and_data(vrm, &new_vma); + /* + * If we established the copied-to VMA, we attempt to recover from the + * error by setting the destination VMA to the source VMA and unmapping + * it below. + */ + if (err && !new_vma) + return err; /* * If we failed to move page tables we still do total_vm increment @@ -1007,51 +1157,15 @@ static unsigned long move_vma(struct vma_remap_struct *vrm) */ hiwater_vm = mm->hiwater_vm; - /* Tell pfnmap has moved from this vma */ - if (unlikely(vma->vm_flags & VM_PFNMAP)) - untrack_pfn_clear(vma); - - if (unlikely(!err && (vrm->flags & MREMAP_DONTUNMAP))) { - /* We always clear VM_LOCKED[ONFAULT] on the old vma */ - vm_flags_clear(vma, VM_LOCKED_MASK); - - /* - * anon_vma links of the old vma is no longer needed after its page - * table has been moved. - */ - if (new_vma != vma && vma->vm_start == old_addr && - vma->vm_end == (old_addr + old_len)) - unlink_anon_vmas(vma); - - /* Because we won't unmap we don't need to touch locked_vm */ - vrm_stat_account(vrm, new_len); - return new_addr; - } - - vrm_stat_account(vrm, new_len); - - vma_iter_init(&vmi, mm, old_addr); - if (do_vmi_munmap(&vmi, mm, old_addr, old_len, vrm->uf_unmap, false) < 0) { - /* OOM: unable to split vma, just get accounts right */ - if (vm_flags & VM_ACCOUNT && !(vrm->flags & MREMAP_DONTUNMAP)) - vm_acct_memory(old_len >> PAGE_SHIFT); - account_start = account_end = false; - } + vrm_stat_account(vrm, vrm->new_len); + if (unlikely(!err && (vrm->flags & MREMAP_DONTUNMAP))) + dontunmap_complete(vrm, new_vma); + else + unmap_source_vma(vrm); mm->hiwater_vm = hiwater_vm; - /* Restore VM_ACCOUNT if one or two pieces of vma left */ - if (account_start) { - vma = vma_prev(&vmi); - vm_flags_set(vma, VM_ACCOUNT); - } - - if (account_end) { - vma = vma_next(&vmi); - vm_flags_set(vma, VM_ACCOUNT); - } - - return new_addr; + return err ? (unsigned long)err : vrm->new_addr; } /* From patchwork Thu Mar 6 10:34:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Stoakes X-Patchwork-Id: 14004152 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33864C282EC for ; Thu, 6 Mar 2025 10:34:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 56983280009; Thu, 6 Mar 2025 05:34:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4C9F4280003; Thu, 6 Mar 2025 05:34:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C935280008; Thu, 6 Mar 2025 05:34:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id ECACA280003 for ; Thu, 6 Mar 2025 05:34:27 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4D095161424 for ; Thu, 6 Mar 2025 10:34:28 +0000 (UTC) X-FDA: 83190767016.03.6A91D5F Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf17.hostedemail.com (Postfix) with ESMTP id EB7964000F for ; Thu, 6 Mar 2025 10:34:24 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=LBUT4bXL; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=NhiKF4oz; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=oracle.com; spf=pass (imf17.hostedemail.com: domain of lorenzo.stoakes@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=lorenzo.stoakes@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741257265; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=53S3I5wh33yuvzRwUMeMC0CFiZy3kRXRqKeBTovpeX8=; b=DtXanz/nKpxvaZgFN+Pnx3sBtDmVq/kp7MTH1h0JrHx+qastnyQ0nsyfFnZt51gASkDA/i 9IY6CjpDg3ty/lkwRR3pjcaTqjywhrIddjj8YkwhyMSjyiLnEGJoRXd2loZ3aiotXC/d3q P7dDHpUoySd99r5fCxN4HlRB/6m8jaU= ARC-Authentication-Results: i=2; imf17.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=LBUT4bXL; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=NhiKF4oz; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=oracle.com; spf=pass (imf17.hostedemail.com: domain of lorenzo.stoakes@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=lorenzo.stoakes@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1741257265; a=rsa-sha256; cv=pass; b=rg576ZbknQQLGpTbrLjBgCKXNdkKiMSYum9oPbNizKFUCNUoRVNnHOVKLXCOMRJ/c2x+J2 FNrLjGoeBBApGPVgvT6pILfhJmdlpS5Nrobc2fx4fCVOK8ibPP3hGTBR23dwNmNMee5UXm f0j/7172r9eydf3+NQvkVqge6oQ1dAM= Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 526ABfhG024533; Thu, 6 Mar 2025 10:34:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=53S3I5wh33yuvzRwUMeMC0CFiZy3kRXRqKeBTovpeX8=; b= LBUT4bXL+yQFh93FHTkND1jHuvwa+5yJxV/tTE7TxMQq6ew/wBPn7tDEPVEzYMG3 ceB8ZZkCEloEgGQ5UeH+7Yplx4rLeST6Jgxj2f9KO6xBbIJNmXTns0LKh8ndffWu SdR3p/oDdNzVIyO3poGS4/nbOYTfqEqSRaN4YPuLiKl/FssQWkthAdGkXFbwAgmo QBI0T/KUExNSxT9oA4JRb3fJJ9S9j+QEbc7a3J56r9azgBiNIbc/FFi5oDKfpjbR TdeBP7GDgry6Clbliud/yeSIq4l8Z9J5NmaP1G3QMLGOoSIQTMv3lxadNNsEqEDA 2CHgc6YO8Sg1x2UEPn0Nwg== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 453u8hhttk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 06 Mar 2025 10:34:22 +0000 (GMT) Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 5268WZMR010893; Thu, 6 Mar 2025 10:34:22 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2177.outbound.protection.outlook.com [104.47.57.177]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 453rpdh6uc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 06 Mar 2025 10:34:22 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=E6Tg6Nv0dwtXhfA+aG5ym6gDaxQ6sZZS8KfQRg+k+sM1bQnFjKZtLAeztAy5cvI/M/hg/ZCdz5qX/f2x3qCG2jYt/7qGT1OF297G8y567EX04YnPA7vxwsqHhwH5HXJk/XhM0tzmw/V+Y0INa/BB+uWnpkow3g8YDjLPfNTljHCeU2GHt7DjyTLsyGgRKlfD2zwFHyKEMv14xnUew5699Y2WvpCtlIdHLh08b7tMRqqJwvm8u5n6AfeJBvOrmqt3Rj6CqmpNl358rVU5u1LT2RLU3n/NHhN5/6hmzqllxx61TmD5B/3AcBmMBl3759A8bW4qTh1XL9FNyGTefzEXtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=53S3I5wh33yuvzRwUMeMC0CFiZy3kRXRqKeBTovpeX8=; b=ZNE+NwjoHFYxUdjY8qI93Qqp9xeUnEkvhy6vbvBFAOjcgao/Cs6wVBT6bWPPtO0yYqBYBN5Pv2OrGTI4oEdoyi9udZFPIb9X795U7NUelBYvJFojioYzi8gIsPJn+MRN+3sdsHGQYERgkj4DtYdj6tjH/Lh7nN4ryBpWCqRAnKaksVpDLZlEX22D4ZlfznHqVszHwGHfszz6qnXpsceU1Jg1CrP/PwW91D594F24PODdYW7UanWLDjSpbVsDfytoVFYQAWYCZS5FYXFWfZcBGR+tFn1BKAf6Ah1vkw5BU9+Fc9/D+KbXOFTxxn4ap4a93pEUVMLET3aWsZUQAPLAGw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=53S3I5wh33yuvzRwUMeMC0CFiZy3kRXRqKeBTovpeX8=; b=NhiKF4ozy3EdTJ3VmtcUbk+qXyphvpw9+EdTbJUFeH0NXc8zHAvB/HS+IwfG37WEQ1u9bnu45orAue8QVhYS/5vTmU95BYxFFQTaKdgrz8/0GHLmEEiqzx079+7+wn+wV59/OLnny9itaP841+nSIyBNTT4Fpf17w2yEgiUUYSM= Received: from MN2PR10MB4112.namprd10.prod.outlook.com (2603:10b6:208:11e::33) by PH0PR10MB4647.namprd10.prod.outlook.com (2603:10b6:510:43::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.20; Thu, 6 Mar 2025 10:34:20 +0000 Received: from MN2PR10MB4112.namprd10.prod.outlook.com ([fe80::3256:3c8c:73a9:5b9c]) by MN2PR10MB4112.namprd10.prod.outlook.com ([fe80::3256:3c8c:73a9:5b9c%7]) with mapi id 15.20.8489.025; Thu, 6 Mar 2025 10:34:20 +0000 From: Lorenzo Stoakes To: Andrew Morton Cc: "Liam R . Howlett" , Vlastimil Babka , Jann Horn , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Harry Yoo , Yosry Ahmed Subject: [PATCH v2 6/7] mm/mremap: refactor move_page_tables(), abstracting state Date: Thu, 6 Mar 2025 10:34:02 +0000 Message-ID: <89f2446f70acd41172dadbb3404db9d95415c78c.1741256580.git.lorenzo.stoakes@oracle.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: X-ClientProxiedBy: LO3P123CA0016.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:ba::21) To MN2PR10MB4112.namprd10.prod.outlook.com (2603:10b6:208:11e::33) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN2PR10MB4112:EE_|PH0PR10MB4647:EE_ X-MS-Office365-Filtering-Correlation-Id: 02d3d8c4-96f1-4d01-28ea-08dd5c9a743a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|366016; X-Microsoft-Antispam-Message-Info: IeV3kaKjHwejKuZia+sgfye9xP8bS8DdDkU8oEPVFBqLFGEmzZOBoBPNQb5oKc3P6r/UG5nXzFwHoe2v0Cs/+i0xeRXWI+P5Rm6YXZ+VH7yoV9jK8y8AFAcw+sNBXBRAaEJFzDob6UvMG39RfNRhE/CPAFore5keLcNaaoh+KU208MUVndsjnax9Qq9JRybrPHTlyedmyxbXo1Hblj88/jQXCcQzTJXENqUefYssTlBfGUSH7ZwXgARkcR0cMmXjpZfsS3U1G/+ReBzkeVR1WzEdfpb4MumUbvyZQJIyvioJ6u887pdW8B/BJvZWL7XyUggjv8D2O7hOWHe3jXvB9pc/+dDp4SFje1B8TB5ssBi4bA1gLNFXgqAlLrmwX5jEteN36LaWNS4iwdR+Mymar3qxQgYWs5+0WLTT6erBeJlCQzAy7yzSTx5vxfq/klQUWkwkpsfsKQAG6/rhvyAWt1YsQSwf9NgmyXCJR7VaHTOxoYyqB1mwmpkGSIukXSOm2HZ8xBGjH9rg0EVTX03IqJZEGUOfGrUoe+h30m6BDnE1sdVym7TZl0wsiIy1Msqd1l4c6TIUOxTkPF3zYki24F1d81PnQL3kiR6uTSqaHOol67Vc1ykc7EwU4pRHG+uahSlvNxdr+k7UtkgsNEjPnVhNFUre6VnZvpLufDuLvv6cbVur4GSABQWT6FoO3i50+lwozNpW1H+yuuQXSF5Lroxnp1KxlCeL2fLEaHJK+LF5cICaiuICE0WtkViiAzgBblrMQFD/CUKbofo7ykj/61x/CB/LHoJcQxcOqlIqfD4KFgpbTqXhiCAyPiXitg8lxPJg9ged3D3vdNShn8zRZpYPBxse/yZOKQw2DlWtNwyzAysNccGdBeSFIwuRS0DWIv8cxjXKKB8n7Nvdbq+lm4tGHfmEwjMgGkLVpPPSo4EjkdA/eRRpafqOm9pb85Zze0pXgXJvpeo9n60CQuYsYCEVqdwy9cJHmREr3Mi+ZQNuxUZMl6W7thj8eqo4sFC1fYqku10+DTZIglmihXOLqqC3bpJgSi10FNSpCgJd+QVpSDy7P3PSRDWmWazh5DD9Sk2AowVtwJPoMSKhTF87b3NcBOA5mTHLZflaE+D00G5QSDF+CUKi477L1ftVxyGbluqaCMP4Q2AiqkpEMx+7zy0ijVc+6WXvMQ3cRcpq78gFmquVv09g5HZPLI2eGl2kVPUk9OTkqeJSkfmz1CbiisG4Zeg2JBb8kSwJr3QX3duR7WXX6db7BGlWOaE5CWiTt9ID/5h2OADZPLlEaMQVlFs6W4IE9blBM3vhSN4tlGc6QXCceBz/RI24iPYx59R2i9AUQLvF45Pt9vGQ0CZhPDdgKfhrLS3NyNj8H20YNdTR9uYAh88Q6Whn0qsWVcdv X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MN2PR10MB4112.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: j063hctU8IAJiaUa9IlOTIBfZ0jEcQTee0meBkdg62IR5lJhJJq+Ezh1N/YwvQBJckbCZDrAnLUD2Ypc0CCWeRmAWCtniAQ1kTnVJBf3BFEHUTaRA4bxchk4qHtdJUyv3LvzuOaAvP8cL7oC0mP5QFw8z+vwL979m3MIcrHap9DgHhWwcLrBXSzbG+pVQqeiC5MwNjw37X0nVpElckKkfDFelkteEtsh0fy01CLyOJIILtKYf0j/BjD/upGh80XVZtQOkHDD0LUE6I+ZnbsySCtM38V6EKDJsm9gLd6y7VVujgAsf+xgjmdsyQ5e9Q2oszsIJWgBUsPFs8ZO/dBzrUhHCItqY3ECDtkfyaF+8tk8m7HOO6MGS1FpotFmaKZ2dD84IW4T89PwFtA4SU1HHMuxpOB7B67OIOw/pZT8wHisWoYYFEEF0U3XJMD4gjFocDYWMDhdpX+ViKxolyVZyzqtusm5LJk7Inon2FlG4FN0dO/q1Ui9Cvng4YMepY00Wq7qmXc40jkAS8Piw6Ock7qVB9Z+QbbEhtmIqI7omGXoQV1aX97J9Pe8a9caS2nntfJB2ONS2CyvG2nKDK8TpRtp8bPV239XhqLKl7nskIFEa3CE8Np8n9QsOHYZwgAo1AVZ7vARm5dLyztRssVbZOADyEQMSwNWkPE1wdQHX7tnV943CH9Q1jJCVpBX5zdhnctiOBlpy+TpxzYTWI+m57feqh55O+wRkiygWn9kEVyuHNwnHZAA5NzdrwGAEF3RSJCl1+zcP+9M1G4UGO+JpndjGHHwWXbHf3G+s3VK6g1YWdHvh3kGizmBqM8Exkz9aCxlSBHjXNu4jRlhBAAssr5bsxlbX5ck1i0tPW1oYPYXL+RhGtoRp5ivyA22TokrORonE3dsfALoGZp679EBEObK8+FtGIM+5N9pvBgqMfivSquQnLEBNruvbG0i+0TAcBkkvOhnWiTbbl5EXy2BrzzYXcguTEXm4FFIu2akxqeW6o6OsInlf7akUfboAZqxPc/Ozr+n6dw5KpB4a3/PDOK1l2DFX8pxpOFhExIiAbc8EJx1Ps5xXnrlsZuMwWBTUGjdqExDuqHQgvn5XFB/0Xwi3ZGqbbW/BmbD0AXOxHJfc6U7Jxo5oNqjjrDJBBYOqxRJVpxJQI30EJIOCsKkgXLPqNnei09E2pO8v0o5jv9uhZ2LXOB/ABfdqxQrqSdPwO/GaLtA+GlUVBksIoISYD8KJf3wYfnzUFnGtP7sj7E3pgyaZIOxEku3IPaqxDCLfiH9Z1gYigUf860K1K5XRl31I1hlqh+iBHTTIwXHsLkocIKtFvfjX8xjONQ3LBBEfTeJPRjrlcxaruTE20wQd9H/jvThwsYrVHObKxBcodxW2J5dH+4a8NkUEOaesD9gXNjic31NDtkJnOspcsE5T5q91SvAcUQ+JBh0/aIDyYEZITANIkd1/SW8++dFYnqF9vMXi14XABSW2ifcG0L96kP0gu/xUZgV9YfS/93Q5XJZ+K4nIlAPSpEIK0xa8tEIU5YzgMWyhctC1jkSyh9MvptVyvjLPl9TOAMRTt4NJXnwa1PIK+GGBpiI6l02YrAlQjmQDHFTjyqo+fpEPyvGJQ== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: uS6S/KBn/0Kuyu9+S3dAFsAAShjHuQIR6IdN/NLTFyJJP34tA0DE0tNUnx9/cbvUBk3SrAdLokdJIApSPCSIxFwfECFstxRFtrlcF9wuzUYWEcuu/1NizBHwXPvFZAOqRS+bwegML8U0kuevUN+vKzuACqYxG6YL3xWC6Hd6gzTf+KWA6LA9N7QprTcOAz+RkXunaoGNi9Em+I2a++PYg/4atblugwV4XFhkWTyupJEjT9+M+nTm5jCKIOwLkhtx+yBtOlxe6S2p6do/anGcZdtp0p3lfr2GtplKd4NBALDdmseZoUlvdVmnuPlhibSphj83+IbHZQhCjxVNvLwU3ftLNOoY3VUXBBIVl6PMNGjjXTy450ltsrKlB+w2aRaWRaV9WB181kvcmQVQXuqvsJMUTRW2432aXkZjM0zEj0baUaYETo1n/nh75joUiyrk14jvXlYWxlhrlyh7lD+dEKTVroilWH/Ny86rTrVBv2MC2qOmlVIkhObWBWG8GXliyh0FXXrhm1ZyLodHm9DLh/RMa1KkAV9HA5N6g88qSIT2FKGjO/uX9S0In8+jcty0V02dEEfERJh5oqWHMJlsKi1FUWk/VrVNi1wIVLmIO0w= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 02d3d8c4-96f1-4d01-28ea-08dd5c9a743a X-MS-Exchange-CrossTenant-AuthSource: MN2PR10MB4112.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Mar 2025 10:34:20.2941 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: c0GH2Ti9WVlgDC/HE2rwWU9Ng1u/irwcecOof94F+JuHKI4rS0d+BDEas+Q2aFl1ZRdr1WMWuS3MNTN0m2p+H8hTN8Gavt8VIzx1uKofuh0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB4647 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1093,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-03-06_05,2025-03-06_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0 mlxscore=0 spamscore=0 bulkscore=0 suspectscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502100000 definitions=main-2503060079 X-Proofpoint-GUID: O4r79k61AZwph2guSg0q2LrFizcX8NY4 X-Proofpoint-ORIG-GUID: O4r79k61AZwph2guSg0q2LrFizcX8NY4 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: EB7964000F X-Rspam-User: X-Stat-Signature: jgicoqfg6kndup1xnhmakur3yhiozk63 X-HE-Tag: 1741257264-344739 X-HE-Meta: U2FsdGVkX18vpDGmx3slCZ8BwMJsM1Mk8cdgpkSvq51+cZYE8ugQQo0oKsjdwyeyWv2KO1j2Ftlef0Tb9p223HqpYScUnDc3Qef76A+AFZ1jN1xMapD1iKn2XUwes0DnSYC+fD/YJ0SSEq04g7WrB4fVD3bQfKLgc2LXUs3oq65MDl21g7TWt+WrzEevtKYyEVafkeC0MBETymh+gcrChWM0vsAv/O7c52H12d07u9kTcL6460T1BDofu5/foaS/FqimyTWKarnxCxEVkE7WUyXKGra1CJsPIolwC53GyFAj6EAofbGS3HxEiuMsHU6PCkLDGLud+GfEhWLG6Ol4KeYAQ4jZjPLO/4QqWz8QOENgvl0NBQL0W7JB9SWlmFf1Z9DCMISNmlGVqLxmNYFAUVPo+POvEqN+muEiGTIZxrP+7tubF7Gc5zLSuv7b++dXPo6wgbUzNh1IMVhGVs6lFcuS8ypB0XUkK2mwXs/5EVxiBjN78gUeadYyrq2Cw48uwL7Vvw4QNAZanF20aY/kYT/qD+S1PILjJjWF8zieorBqZKBS9UzU1Vz/K4nwkgZ6eLQs8HWm+a9TTiH4xYoXSOflk7VkNBmDAE4nlde3pJpPTUQ7hJGixZ/BlhrCmIMkoTseLbqXBQ8H2YVsjMw0MDvQlipJvDjEeLM0KwPCuq6pDJ8OXyT6hPOB11hzNR8Ud3upQVNBwvK/x46gyWndd8b8EuX1tH7dXTxAF/IMtAavtof2EvTGRDSG3ieljulVezpJfjiUVwzckijdUIUOeYz86p556JUJdIQK7xWXnynN4YInYOw4SCe3JiV4Us+ds6wL3ox+BhyPmxp0kSTAyNZtPNU9hpXNoulaAGtGvwGMYZNv3MCIoSEEsPlxzWq2FHYJBNvJ7Wbod8iMkmaesSjm3ChgDOI5XBusbRL89B3YlYWHfBTadZh0mqMG1lQV12rDxNe+OH2g8LBmpcv Jau9G4C2 K2py8bOGgO4Cw27lOFJHTcYpI4+gK9gq9vtvpl8g5gUuINITHi0fnvo4kcofCSLQq2mBR8OBa312A1j7XEccsff2N9EC84z21OLnUwi290xQ4jTFeTA1RJ2JTBcZK1sY3Lln1etPDJzKKdIPnphe9XgTpnLZ+dTdJJlc+2ZhvfPna2/9vpFS+bwszC5SUCWQ9208dAxSZIv1874Q8doaAkrzSiDO9OcH0qtSyvt4/cIGMFyVHenRrfWartyX/J7pWY2U9HbsFSs1tWr8C0X5JURy4M6Oewz7SBiNqANHo897NQRC5qZHZkrjCBVJNCNlQKUIbV0CI9fE9An+LQGunTTIbFsaMayUPgULD7hI+NyFaYLzVX7NO5y4ke7uGaXuuBOyxQY7S4iSkHyEGNPSOSEUlQuRGqEqESQyXXuOsnNfFejSNFDVHpNveMS+2IhJ1fyo9v8/d/0W8dQ5oBdbi6bkA45fXC8XmHrwAaBMmZ/T02+Hp8+0BjUBP53wsFuye+dYHAUTKQ/W6boXJwiJLRU7085FtNAo/FDCO3McUlaw3/KONvN4yUZlvKrIcBpIUaHbWaP/SEE6aVNYmf0Z6njmyFIWawbMBGH29jA0kuiKOuBzoyclGZJaZIUV95dIv8YynKt2Ei1e+LZz2UkRN0F8n2ay71CV42LK39UUx8o2Lm8++G6XHw6fbUqDco3udri98 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A lot of state is threaded throughout the page table moving logic within the mremap code, including boolean values which control behaviour specifically in regard to whether rmap locks need be held over the operation and whether the VMA belongs to a temporary stack being moved by move_arg_pages() (and consequently, relocate_vma_down()). As we already transmit state throughout this operation, it is neater and more readable to maintain a small state object. We do so in the form of pagetable_move_control. In addition, this allows us to update parameters within the state as we manipulate things, for instance with regard to the page table realignment logic. In future I want to add additional functionality to the page table logic, so this is an additional motivation for making it easier to do so. This patch changes move_page_tables() to accept a pointer to a pagetable_move_control struct, and performs changes at this level only. Further page table logic will be updated in a subsequent patch. We additionally also take the opportunity to add significant comments describing the address realignment logic to make it abundantly clear what is going on in this code. Signed-off-by: Lorenzo Stoakes Reviewed-by: Vlastimil Babka --- mm/internal.h | 46 ++++++++++++-- mm/mmap.c | 5 +- mm/mremap.c | 172 ++++++++++++++++++++++++++++++++++++-------------- 3 files changed, 171 insertions(+), 52 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 7a4f81a6edd6..a4608c85a3ba 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -24,6 +24,47 @@ struct folio_batch; +/* + * Maintains state across a page table move. The operation assumes both source + * and destination VMAs already exist and are specified by the user. + * + * Partial moves are permitted, but: + * from_vma->vm_start <= from_addr < from_vma->vm_end - len + * to_vma->vm_start <= to_addr < to_vma->vm_end - len + * + * Must be maintained. + * + * mmap lock must be held in write and VMA write locks must be held on any VMA + * that is visible. + * + * Use the PAGETABLE_MOVE() macro to initialise this struct. + * + * NOTE: The page table move is affected by reading from [old_addr, old_end), + * and old_addr may be updated for better page table alignment, so len_in + * represents the length of the range being copied as specified by the user. + */ +struct pagetable_move_control { + struct vm_area_struct *old; /* Source VMA. */ + struct vm_area_struct *new; /* Destination VMA. */ + unsigned long old_addr; /* Address from which the move begins. */ + unsigned long old_end; /* Exclusive address at which old range ends. */ + unsigned long new_addr; /* Address to move page tables to. */ + unsigned long len_in; /* Bytes to remap specified by user. */ + + bool need_rmap_locks; /* Do rmap locks need to be taken? */ + bool for_stack; /* Is this an early temp stack being moved? */ +}; + +#define PAGETABLE_MOVE(name, old_, new_, old_addr_, new_addr_, len_) \ + struct pagetable_move_control name = { \ + .old = old_, \ + .new = new_, \ + .old_addr = old_addr_, \ + .old_end = (old_addr_) + (len_), \ + .new_addr = new_addr_, \ + .len_in = len_, \ + } + /* * The set of flags that only affect watermark checking and reclaim * behaviour. This is used by the MM to obey the caller constraints @@ -1537,10 +1578,7 @@ extern struct list_lru shadow_nodes; } while (0) /* mremap.c */ -unsigned long move_page_tables(struct vm_area_struct *vma, - unsigned long old_addr, struct vm_area_struct *new_vma, - unsigned long new_addr, unsigned long len, - bool need_rmap_locks, bool for_stack); +unsigned long move_page_tables(struct pagetable_move_control *pmc); #ifdef CONFIG_UNACCEPTED_MEMORY void accept_page(struct page *page); diff --git a/mm/mmap.c b/mm/mmap.c index 15d6cd7cc845..efcc4ca7500d 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1694,6 +1694,7 @@ int relocate_vma_down(struct vm_area_struct *vma, unsigned long shift) VMG_STATE(vmg, mm, &vmi, new_start, old_end, 0, vma->vm_pgoff); struct vm_area_struct *next; struct mmu_gather tlb; + PAGETABLE_MOVE(pmc, vma, vma, old_start, new_start, length); BUG_ON(new_start > new_end); @@ -1716,8 +1717,8 @@ int relocate_vma_down(struct vm_area_struct *vma, unsigned long shift) * move the page tables downwards, on failure we rely on * process cleanup to remove whatever mess we made. */ - if (length != move_page_tables(vma, old_start, - vma, new_start, length, false, true)) + pmc.for_stack = true; + if (length != move_page_tables(&pmc)) return -ENOMEM; tlb_gather_mmu(&tlb, mm); diff --git a/mm/mremap.c b/mm/mremap.c index df550780a450..a4b0124528fa 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -580,8 +580,9 @@ static bool move_pgt_entry(enum pgt_entry entry, struct vm_area_struct *vma, * the VMA that is created to span the source and destination of the move, * so we make an exception for it. */ -static bool can_align_down(struct vm_area_struct *vma, unsigned long addr_to_align, - unsigned long mask, bool for_stack) +static bool can_align_down(struct pagetable_move_control *pmc, + struct vm_area_struct *vma, unsigned long addr_to_align, + unsigned long mask) { unsigned long addr_masked = addr_to_align & mask; @@ -590,11 +591,11 @@ static bool can_align_down(struct vm_area_struct *vma, unsigned long addr_to_ali * of the corresponding VMA, we can't align down or we will destroy part * of the current mapping. */ - if (!for_stack && vma->vm_start != addr_to_align) + if (!pmc->for_stack && vma->vm_start != addr_to_align) return false; /* In the stack case we explicitly permit in-VMA alignment. */ - if (for_stack && addr_masked >= vma->vm_start) + if (pmc->for_stack && addr_masked >= vma->vm_start) return true; /* @@ -604,54 +605,131 @@ static bool can_align_down(struct vm_area_struct *vma, unsigned long addr_to_ali return find_vma_intersection(vma->vm_mm, addr_masked, vma->vm_start) == NULL; } -/* Opportunistically realign to specified boundary for faster copy. */ -static void try_realign_addr(unsigned long *old_addr, struct vm_area_struct *old_vma, - unsigned long *new_addr, struct vm_area_struct *new_vma, - unsigned long mask, bool for_stack) +/* + * Determine if are in fact able to realign for efficiency to a higher page + * table boundary. + */ +static bool can_realign_addr(struct pagetable_move_control *pmc, + unsigned long pagetable_mask) { + unsigned long align_mask = ~pagetable_mask; + unsigned long old_align = pmc->old_addr & align_mask; + unsigned long new_align = pmc->new_addr & align_mask; + unsigned long pagetable_size = align_mask + 1; + unsigned long old_align_next = pagetable_size - old_align; + + /* + * We don't want to have to go hunting for VMAs from the end of the old + * VMA to the next page table boundary, also we want to make sure the + * operation is wortwhile. + * + * So ensure that we only perform this realignment if the end of the + * range being copied from is at least page table aligned: + * + * boundary boundary + * .<- old_align -> . + * . |----------------.-----------| + * . | vma . | + * . |----------------.-----------| + * . <----------------.-----------> + * . len_in + * <-------------------------------> + * . pagetable_size . + * . <----------------> + * . old_align_next . + */ + if (pmc->len_in < old_align_next) + return false; + /* Skip if the addresses are already aligned. */ - if ((*old_addr & ~mask) == 0) - return; + if (old_align == 0) + return false; /* Only realign if the new and old addresses are mutually aligned. */ - if ((*old_addr & ~mask) != (*new_addr & ~mask)) - return; + if (old_align != new_align) + return false; /* Ensure realignment doesn't cause overlap with existing mappings. */ - if (!can_align_down(old_vma, *old_addr, mask, for_stack) || - !can_align_down(new_vma, *new_addr, mask, for_stack)) + if (!can_align_down(pmc, pmc->old, pmc->old_addr, pagetable_mask) || + !can_align_down(pmc, pmc->new, pmc->new_addr, pagetable_mask)) + return false; + + return true; +} + +/* + * Opportunistically realign to specified boundary for faster copy. + * + * Consider an mremap() of a VMA with page table boundaries as below, and no + * preceding VMAs from the lower page table boundary to the start of the VMA, + * with the end of the range being at least page table aligned: + * + * boundary boundary + * . |----------------.-----------| + * . | vma . | + * . |----------------.-----------| + * . pmc->old_addr . pmc->old_end + * . <----------------------------> + * . move these page tables + * + * If we proceed with moving page tables in this scenario, we will have a lot of + * work to do traversing old page tables and establishing new ones in the + * destination across multiple lower level page tables. + * + * The idea here is simply to align pmc->old_addr, pmc->new_addr down to the + * page table boundary, so we can simply copy a single page table entry for the + * aligned portion of the VMA instead: + * + * boundary boundary + * . |----------------.-----------| + * . | vma . | + * . |----------------.-----------| + * pmc->old_addr . pmc->old_end + * <-------------------------------------------> + * . move these page tables + */ +static void try_realign_addr(struct pagetable_move_control *pmc, + unsigned long pagetable_mask) +{ + + if (!can_realign_addr(pmc, pagetable_mask)) return; - *old_addr = *old_addr & mask; - *new_addr = *new_addr & mask; + /* + * Simply align to page table boundaries. Note that we do NOT update the + * pmc->old_end value, and since the move_page_tables() operation spans + * from [old_addr, old_end) (offsetting new_addr as it is performed), + * this simply changes the start of the copy, not the end. + */ + pmc->old_addr &= pagetable_mask; + pmc->new_addr &= pagetable_mask; } -unsigned long move_page_tables(struct vm_area_struct *vma, - unsigned long old_addr, struct vm_area_struct *new_vma, - unsigned long new_addr, unsigned long len, - bool need_rmap_locks, bool for_stack) +unsigned long move_page_tables(struct pagetable_move_control *pmc) { unsigned long extent, old_end; struct mmu_notifier_range range; pmd_t *old_pmd, *new_pmd; pud_t *old_pud, *new_pud; + unsigned long old_addr, new_addr; + struct vm_area_struct *vma = pmc->old; - if (!len) + if (!pmc->len_in) return 0; - old_end = old_addr + len; - if (is_vm_hugetlb_page(vma)) - return move_hugetlb_page_tables(vma, new_vma, old_addr, - new_addr, len); + return move_hugetlb_page_tables(pmc->old, pmc->new, pmc->old_addr, + pmc->new_addr, pmc->len_in); /* * If possible, realign addresses to PMD boundary for faster copy. * Only realign if the mremap copying hits a PMD boundary. */ - if (len >= PMD_SIZE - (old_addr & ~PMD_MASK)) - try_realign_addr(&old_addr, vma, &new_addr, new_vma, PMD_MASK, - for_stack); + try_realign_addr(pmc, PMD_MASK); + /* These may have been changed. */ + old_addr = pmc->old_addr; + new_addr = pmc->new_addr; + old_end = pmc->old_end; flush_cache_range(vma, old_addr, old_end); mmu_notifier_range_init(&range, MMU_NOTIFY_UNMAP, 0, vma->vm_mm, @@ -675,12 +753,11 @@ unsigned long move_page_tables(struct vm_area_struct *vma, if (pud_trans_huge(*old_pud) || pud_devmap(*old_pud)) { if (extent == HPAGE_PUD_SIZE) { move_pgt_entry(HPAGE_PUD, vma, old_addr, new_addr, - old_pud, new_pud, need_rmap_locks); + old_pud, new_pud, pmc->need_rmap_locks); /* We ignore and continue on error? */ continue; } } else if (IS_ENABLED(CONFIG_HAVE_MOVE_PUD) && extent == PUD_SIZE) { - if (move_pgt_entry(NORMAL_PUD, vma, old_addr, new_addr, old_pud, new_pud, true)) continue; @@ -698,7 +775,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma, pmd_devmap(*old_pmd)) { if (extent == HPAGE_PMD_SIZE && move_pgt_entry(HPAGE_PMD, vma, old_addr, new_addr, - old_pmd, new_pmd, need_rmap_locks)) + old_pmd, new_pmd, pmc->need_rmap_locks)) continue; split_huge_pmd(vma, old_pmd, old_addr); } else if (IS_ENABLED(CONFIG_HAVE_MOVE_PMD) && @@ -713,10 +790,10 @@ unsigned long move_page_tables(struct vm_area_struct *vma, } if (pmd_none(*old_pmd)) continue; - if (pte_alloc(new_vma->vm_mm, new_pmd)) + if (pte_alloc(pmc->new->vm_mm, new_pmd)) break; if (move_ptes(vma, old_pmd, old_addr, old_addr + extent, - new_vma, new_pmd, new_addr, need_rmap_locks) < 0) + pmc->new, new_pmd, new_addr, pmc->need_rmap_locks) < 0) goto again; } @@ -726,10 +803,10 @@ unsigned long move_page_tables(struct vm_area_struct *vma, * Prevent negative return values when {old,new}_addr was realigned * but we broke out of the above loop for the first PMD itself. */ - if (old_addr < old_end - len) + if (old_addr < old_end - pmc->len_in) return 0; - return len + old_addr - old_end; /* how much done */ + return pmc->len_in + old_addr - old_end; /* how much done */ } /* Set vrm->delta to the difference in VMA size specified by user. */ @@ -1039,37 +1116,40 @@ static int copy_vma_and_data(struct vma_remap_struct *vrm, unsigned long internal_pgoff = internal_offset >> PAGE_SHIFT; unsigned long new_pgoff = vrm->vma->vm_pgoff + internal_pgoff; unsigned long moved_len; - bool need_rmap_locks; - struct vm_area_struct *vma; + struct vm_area_struct *vma = vrm->vma; struct vm_area_struct *new_vma; int err = 0; + PAGETABLE_MOVE(pmc, NULL, NULL, vrm->addr, vrm->new_addr, vrm->old_len); - new_vma = copy_vma(&vrm->vma, vrm->new_addr, vrm->new_len, new_pgoff, - &need_rmap_locks); + new_vma = copy_vma(&vma, vrm->new_addr, vrm->new_len, new_pgoff, + &pmc.need_rmap_locks); if (!new_vma) { vrm_uncharge(vrm); *new_vma_ptr = NULL; return -ENOMEM; } - vma = vrm->vma; + vrm->vma = vma; + pmc.old = vma; + pmc.new = new_vma; - moved_len = move_page_tables(vma, vrm->addr, new_vma, - vrm->new_addr, vrm->old_len, - need_rmap_locks, /* for_stack= */false); + moved_len = move_page_tables(&pmc); if (moved_len < vrm->old_len) err = -ENOMEM; else if (vma->vm_ops && vma->vm_ops->mremap) err = vma->vm_ops->mremap(new_vma); if (unlikely(err)) { + PAGETABLE_MOVE(pmc_revert, new_vma, vma, vrm->new_addr, + vrm->addr, moved_len); + /* * On error, move entries back from new area to old, * which will succeed since page tables still there, * and then proceed to unmap new area instead of old. */ - move_page_tables(new_vma, vrm->new_addr, vma, vrm->addr, - moved_len, /* need_rmap_locks = */true, - /* for_stack= */false); + pmc_revert.need_rmap_locks = true; + move_page_tables(&pmc_revert); + vrm->vma = new_vma; vrm->old_len = vrm->new_len; vrm->addr = vrm->new_addr; From patchwork Thu Mar 6 10:34:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Stoakes X-Patchwork-Id: 14004154 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E84FAC282EC for ; Thu, 6 Mar 2025 10:34:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0753D280008; Thu, 6 Mar 2025 05:34:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 02C51280003; Thu, 6 Mar 2025 05:34:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD528280008; Thu, 6 Mar 2025 05:34:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id AB660280003 for ; Thu, 6 Mar 2025 05:34:33 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id CFCECA9DBE for ; Thu, 6 Mar 2025 10:34:33 +0000 (UTC) X-FDA: 83190767226.27.649B087 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf17.hostedemail.com (Postfix) with ESMTP id 564F94000F for ; Thu, 6 Mar 2025 10:34:30 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=oz645Z5T; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=LvNxAtpi; spf=pass (imf17.hostedemail.com: domain of lorenzo.stoakes@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=lorenzo.stoakes@oracle.com; dmarc=pass (policy=reject) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741257270; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GGAIFI9CI0x0QrLT3hB7iuFb+RRoFDcK+Z7dwHZUaM4=; b=dmW4CxDHnUTVvNSY/ptDbZi73JMsy05yCe+TeqKBq6bTt+YWmvmXEs5ngA+DuN7vRrVA6P Y5ImHStZ6ydufjUXk+jSuWdgPdCHAwnd8mWT/+vsH5ll2Jciy/SV56AWqTpKwQbfED73mq NxxlrQcgQEmdkMKVePG7mPdIyh/DXOg= ARC-Authentication-Results: i=2; imf17.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=oz645Z5T; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=LvNxAtpi; spf=pass (imf17.hostedemail.com: domain of lorenzo.stoakes@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=lorenzo.stoakes@oracle.com; dmarc=pass (policy=reject) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1741257270; a=rsa-sha256; cv=pass; b=7ZSsUoUDZreGQoxXVM1KkeUAfJUkZX13xh1Xl9WtWUZ+s6sLH+gCJFkRu+a9H+944wWoXO f6Cv5JxA13FwQlr4cIRySwE75ZKNq7XjVhItHp+4zsE2tQCIb/7RWqzARxeIcZp0/LezBK VD8UgvM66uo3Heei9wWTFWZP8NMjAP0= Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 526ACWWE025001; Thu, 6 Mar 2025 10:34:27 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=GGAIFI9CI0x0QrLT3hB7iuFb+RRoFDcK+Z7dwHZUaM4=; b= oz645Z5TtXip4oyT83UM8aOPJMX1L2111R7/wLUj4oZzCyW6oUgFmNAdCiVfQRzm gAmqLOGZaBWtOZr0ba0zvFXzFHWlLs/kVPfu15gltY56F9HJ9BNK4iGTWSdbHP1r 8f00/J4NPdUpa7/UVmPQ1cw/eoaXKbKRB6wZJ0FEelbuVxAS+rSFe8ee7w7/i42n AcGtL+B9p6QVZ8diIyhAshiqtnqk+lLny2ZGJc+doAByd1TDiLKOI2B2YR3ILVLZ 4IHFhQrskoxjf2z+CrLXz7UIvLd84c2p9B13TRMQh3TKzaelXnegyqAmowF2Jxtm rcJUCArM8diADHk5LT8iMQ== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 453uaw1sab-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 06 Mar 2025 10:34:26 +0000 (GMT) Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 5268xmFg010942; Thu, 6 Mar 2025 10:34:25 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2177.outbound.protection.outlook.com [104.47.57.177]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 453rpdh6we-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 06 Mar 2025 10:34:25 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Y+n81w1jjJ0UOMkpsIjneXfZB2DbPZfSuI2HLpcxVNP6th7hc3AcwJ/62DW1lQQX4Yo2thiAhB/XNtnATMOL85jWa40cbusl/vl2ajlG92rtO+ECbZ1zGT9FS9FymOe2HyyzO3xn0zxAMGmCBZOHp8JooypP050j95ZuVlfYQ4pThv2WTv947EArHodQD/KfwqP6Stt115+JPbcVoUDL/rCilKR6eubHy5jOp8ZXwPuMU+dJasJkS9Hd1Yf+g6lePXgebEXk3NjiHIOYQ3kOS0cPhWpoZROFxm8QbzoGlpqBbKbw3LyKJqrS276/5AkfgpiQjemIw3ZFYOhONXvBtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=GGAIFI9CI0x0QrLT3hB7iuFb+RRoFDcK+Z7dwHZUaM4=; b=S8xNYOXENaH7M2PziRzgOw0S/QCAg4k4WWkIC2IzhWdUMoq5fRsn1tMU8p2rMj/7/nokusbLZGgttRjAFCK1OlJdh/NAXEiSJHCQfhMWFhpQXgRETXuokFX0DRusQEwJiuUBklt3nlc+jueAp0/ClhIIrUqdEvq8StDM/bW1rMl71pz6gJNe7UgSgbzipt2XslfQ6Js5AwWHWvTBeexRKKmL1bP3aSdB7y+D56MQlmh+z38+eV5h2IvTHjNYeR7gEYD9qhckaI7I/3oiDZRKQq72JdClV/n7AJfiGMPgSXa7ldHoz8kT2Il41wB+eUoun9ETpdMAOI/LG96Yb+r2mg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GGAIFI9CI0x0QrLT3hB7iuFb+RRoFDcK+Z7dwHZUaM4=; b=LvNxAtpiYnmK1nLetvfpGXp2BIQY3sn9Cip1XgevDAPL/y8qd15KLlFsp3ZzZmR+KO1YbDBwrTeI6ym3+eFCtSHTrs/s8rDntljjQCBotRuy5JkeCI7clCFENGwy9ILOX2F6lfjaECFaStyKJTcjs+lMeXBIVSAXj6Wwutr1cpM= Received: from MN2PR10MB4112.namprd10.prod.outlook.com (2603:10b6:208:11e::33) by PH0PR10MB4647.namprd10.prod.outlook.com (2603:10b6:510:43::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.20; Thu, 6 Mar 2025 10:34:23 +0000 Received: from MN2PR10MB4112.namprd10.prod.outlook.com ([fe80::3256:3c8c:73a9:5b9c]) by MN2PR10MB4112.namprd10.prod.outlook.com ([fe80::3256:3c8c:73a9:5b9c%7]) with mapi id 15.20.8489.025; Thu, 6 Mar 2025 10:34:23 +0000 From: Lorenzo Stoakes To: Andrew Morton Cc: "Liam R . Howlett" , Vlastimil Babka , Jann Horn , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Harry Yoo , Yosry Ahmed Subject: [PATCH v2 7/7] mm/mremap: thread state through move page table operation Date: Thu, 6 Mar 2025 10:34:03 +0000 Message-ID: <1d85814f3a73eaa241c554b3e0f751f3e267a467.1741256580.git.lorenzo.stoakes@oracle.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: X-ClientProxiedBy: LO4P265CA0250.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:350::12) To MN2PR10MB4112.namprd10.prod.outlook.com (2603:10b6:208:11e::33) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN2PR10MB4112:EE_|PH0PR10MB4647:EE_ X-MS-Office365-Filtering-Correlation-Id: e983fdab-03c9-4b8c-6167-08dd5c9a760e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|366016; X-Microsoft-Antispam-Message-Info: Y85LrA9ugxZ3GM+d7Hqv8l3WLzz64+B6NH2oYqA3otjofLqQkrp/B3Yik6AQdcc8MSCoxpD0WuBoRWWLAIdIxD7js6MSAHVoq/MMZTdzDPVXUfzpw3q8Ltanim7W/byCnIbBNpmPxauKuOyC9athKM3LQZkeoWRK+nrHJLF0xrhLDZZQRo/ae55R4vMnmC/O6Qj2ODo/fwGnPj05U/W1gqKcGzat26KQ80cBfI/thzt+R1I8o0g4YAAuBN2YwfO4DkxthKN1WCLJ3ELqBHcAJ6IhbvBC2loQ8MOsd55jyuFPfPqvhtVfnFnKIdkYKgRwvKwtBqOi1OAcwcwqmdGroPKet3nLgTmgtHgGQ4n8yTrjJhIOlnPiGzpQbzWQucI+k2mwriDyStjD9r8wqMjGuAtLd5OYI9meW0lvia+C56zI1tw7JMjbr1fFTO3Dfqe5DEytntBFPGoeA1pMTkOkR0deGZ4x9mDW7F+beEQ0ew3xP9911X9U6eEHhS9OYcRtAqoEL5FlNSqE26JwPn0JDNARsEjcmhzvlwdNqxg+DSL0Eb2wxrFbwnp20OCRGYJKR/mQDAqGJN17kiJAVR33Bingh5GEQ9lshNwKWY4WlMzCX9WITmMrlYokm87mR18jP0YnnlOMX3nhAGfGmF7OhlR2iJOxwysbtdo0O8roOXWpBwSg0lSHy3NnHbmyB8rCEicld80VZQhHTINtjWZaX3ke6K2SClXwVkWmMENkeNjCKnFsYEE4ABv1Q3CDquoc1gs7lEC0ZipPBLkTzTJy6hbcMGaMWrJWI3th6GKh1dmR5RrDP+Izb/cjJzIfjE3pJkEleh+M0v6g5z9vXl8OLQf2bMRrlnU9CeqDPB6iaX0oIX0ctJTSMKlowHFb9sikNzXKikFsMFXLB+wER9XDCiNlTaTGU3bWOGLjPjeBCbS3mI9M4Mn6Vj9BSVeQoiyPcqoZHGSR8NMJ6FYcpp99dIn6KF8+Vree6tzBL1EX9NphOkEE8wjbdSTMxhtC9VbQkdxCk6M6QhEdh1SyCrbJV4Ng2G0VynsWOLLcRoUP1qKUQ1IfgCWQO2+NRY2DD/kT7laoydv2+D+iH00EAwi0KQBANu11rX6JiyyfV7rCUS9bsWUCIUgnVycjVHiiHQ8Jz9wE5u4oT+SCi5I0Agg9g9inHQGHvqitGBPcT8lyKNGrgylOwthiXnVZD6VZTidsL24HUmdxNLzaYnkWdWnFy6z2ZuMepvUUhS+KAi5jMEoNDegJB61RM9lyh3K+ieOAwofG7sPdCBgxkH23+zjZa3yVIwNCrczhyLm7VuyKwlhBH1XwW8tB68L9KRCH5SsgWmC6dhX9hOFQnwiPu38e1Wr268Ohcm8gDqNXv4mhNzZ04diyQTNjxOb5H+q2D6H4 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MN2PR10MB4112.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Ji5Rqn7rZv3MDVKDk9Q9wZjNTK+yllLJ8lqfvSMaGy00Wxs6zCkOOCgv6O5xqihmmilJvMCepRgBF99OU/hAtyxWAuBWmhxneRqDGgU4tw5q8fYUt9NVm2WYREjaAlFE0F8PHOe1xWsk06Ig6mknk/We1I5jesdehUcMoIYEHUoi2NNhTcM9TFlTxg7NpBFtsjYIgm0SrRqZ2OHXuC8jQjBIJth9xpe3ZvG/6JvLUoMN0y9t4g9vQMFVOVU82EK61JoLBscSlKxXc+zXzmUu4ReZLLWMa+Bb0tWvMMmUlFehyy7FSm4SGeO+laEx5BGdPkFU+eBkN2x2aAlDIjbnU/o9QBQ+ikH7oICHKy4Rb6WVUJnBJK8mmrHVtgyCL4ZfVtj+cs2i8D3nEwerR/QC6DqFoRHB/AiY0zxj1QX5+qS2Inby1APUmZVJZAclYILEfZBVl+ailb+a48UqkqZtKrCH/fMxikbBpsybMxSUt4bKuVEFdQ9/kyTTE1gYrIs/5XPhf/7nkNNPwPgYFvx/PPyoqZi1LFnVD66/j6pugeNZC83OfTmjbiNjtrTBK8qQC+bXPKMUXWQwQXZlEqQIrTnXrgw0EKCm3SQdUIYiPa4tUddnU3DMsgweZhnp9p9FvTro6sRouwt0DUODAdb5BbxcdwR86QehTs8jlJsi93YvFUGDgTot9Ti0bHaV0FmxarBqAb2ij6icW5HAp15Da/VPUoWw5sUn8JM7HXME0HPOajN0pHyAyqja5GRMWCFtrA0BCOlIx95DXznDlEGNtERH8bgmK7+YQi8wfdrSKZrAImLMVSBZh4ho7zUfdTg2SUn0jmnix4nJtRJzeaNonJ98np3/VWE18KtGXfz2eiljLhao8HpBGMH+ctZfX/h6I/luzl9qrtXVo7hRmCobw5vovOIPKImC2qeLZqgywkAf6yev8ztG6P1mS1H8HrSiGucV7PFaeg1wYQewwYoiieQQJuDnHuIjKweQOqQA3X1cAj2w+I5ZiawoYuTHhxQXVIGayMMhe93QPba0W3mnzgfO3/zOtxY4eaxGymVEG5aynrlRYtHGGP6rIyfsPr+LEPAjZ+IlyIwk1sUWziquRQ41scoFVzFMDWjWLahk5zHrSqKuCsEJr10xrlId7stFAcEbd/T5SGqPVs94M8wUufUjcXGM5hCjHlj07Gqpv+lodp03kaDDqgHyCxi6q6eh5mE+8DVnRPGa/galPXQtLM7Xp4hJuGtr8bvh3SZoJnrFXjVONAKmZffHqpeorq8cupJUJsBd+UFrkr0z0oVnlrwwSMu5fYukvNg/b26lWet7Y2RTKqrrNdn8hbqy15TxbMA3epfGUG4MA7MY9vhAw/RgT1jDoZaLUKP4r2AzbhuedRreBzruORBR7s6EhJmVqYkcDe03/NH1II/u9ykcYh38u9IcxyPJNBsFCM8dlpHayMfsYRWePyJZulSHwBRAf7WuK3Np8MWDWlsi1M/nKEc2GTQX27g8SRKMpG5HYmW7V7sIIDoBSrf8V9xp3XXnLNYZAJsmNBJY8JEeJyXcskdw/b0RvMk8hyeFRy78RiDh4POVGPwY3oEM7GbDWwikqJFotyYb5WMqihOD76HDkw== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 9iNz6dKC+HxGygpAcYwC5NznZzuCkDonMVQymQjD0PmgUtGMMAL7nhcxE7LoVPa2l9y+oXi01oC6KJLAN36cK7DghrLBrCoDQ0EaMvmVH0T9Wix7rqX3+VkOaY2fKuUnDDL5QXK7rCUXoGAZT0AO3yVh5DZpRULyW9qnKjUuoQ1zBuudiIcAhVlTPbKN3z/6T6iMZ7H1z77D4BIdUQenZaTwR1U2NuDQv1sVjozRrf3dAmr3Mlas+2GPfGQtsIye234rbaQw8FxIDs3a/vVecHi+TcGTDg/9RqrDThof2dei3Qtt/UnYaCG7JNFZVSKEbdU3w+rQuXRrmsCTG/EA4MVNi34+zuJSpbd6+BwUKnYVfdL6Hl4XtgJgNZww+HJwBE0lXpZ0WmJEUAD3eM9vDZ/eO6rsw+BGFpMCldowBJqzGLtSYR3UYMitSwqzhpMqx/d5eOlBRnqnHPyR/IyIMw5Vp57QD1QzawgZENnQq+xDFhbH/Q7tsiUS1mNqVkFLCbvs1LuYz77Pq0KsOWcKoWkS96kibpNaXhztzsaJ9IdQXPzHNy/aaIiPVwLELhcDft0e5nAKg7W7OXb2nWJi7y90ATBJP0z16pyXHmzBQGQ= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e983fdab-03c9-4b8c-6167-08dd5c9a760e X-MS-Exchange-CrossTenant-AuthSource: MN2PR10MB4112.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Mar 2025 10:34:23.4498 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KXdvtoOgivQYhwHbYx592r2T1wy4ASrPX9UUbuhCEEfEHXcH/NJ4FZzPbtvIbctn3CSqLoDHEiYX8rCdCsNxYMQYXSnrihJ/7QLqebJySSM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB4647 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1093,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-03-06_05,2025-03-06_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0 mlxscore=0 spamscore=0 bulkscore=0 suspectscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502100000 definitions=main-2503060079 X-Proofpoint-GUID: xj2vhBG5Vwe91pZ8gdrD59YOSILViFJ9 X-Proofpoint-ORIG-GUID: xj2vhBG5Vwe91pZ8gdrD59YOSILViFJ9 X-Rspam-User: X-Rspamd-Queue-Id: 564F94000F X-Rspamd-Server: rspam09 X-Stat-Signature: 1iw6448ib4g9kbiqp56aqtjersy9q6uo X-HE-Tag: 1741257270-66453 X-HE-Meta: U2FsdGVkX1+ZcxPeok/P9HFM4ekKzLhX+q0aRtaDtuESuBcVOPzNx5r/BleDM+i1aoywyfTEeULNkio+gjwyxktgUBDCUIB840S7sStVaagDlPVFqERX9Q6JQekwQBuRdXJJOFpu0UWcLSWZCttknS8Kgjn1giUX38CjWVYTrsOfcNVURX8sU4VlqpPvkFWxlRY8+YsQ5cvzss0aG/FexymVInJB/YJ7g9J8bXCBQL1t9GmGWLL78FmrIzF7Cb+xiWp6NBj+Add3v3/gMjqQoPH/twgKcv+8RyH4Y756DVjunuZz8KLSKsUcIMbINeKgxGFsiAvcAJqlr8/V485xL7VZexSBjpXcmS8hZu9ejVfiTqZuzxBVmbPMQIW/xZvMoCLSl4I2iNaN25JFy9so/MVoVYfFQQ0OZXlPZQXqspatLRoDcCEbSbBTk2syW84N0TrDIgpTdzTFWWPGoSZGY8BTAK2gLY42eGV8BjE3SM0yEM5tme1pzmEhLjIvPczjt26jtd1ffDQvjG844i0MB6iC6Wa37SC4FeO1qsbHF/M6pKmA1CGYlWHQ6LNnW8yFoeG6WylZQjbeW9QcNtwHZ3nYUDZekvrTP8aRzAI8Jx/EVvNoIS82HuXc+9+nzJakXXd2PM58aRMnmoBQ9WgZUV41ubmGDeILpovbGzqXOuw9PQR15NoPZRjPUfF/em3gjUcvYWj+qMcv4JoinRqxn5UKmTpFP0NxXpyefMZkr0p2Pfa8uQtbdm0RbXTa3IU9/zPHEN/7mj1TyiTrfpv6D6ZeyBXkHWD1MKNpcgbUK3B89G3DIfVIJnaiNwhr2hMZ54BJYj5J4V5pV9p4S9RHzAgB3XH4TGfdbGhb18YwQELNt+7pnDBgQ5Vn5SjMdceTlARFi6vQM9iBQrF3viVn7l7GK2Yz8FBW0Apkqj2SoPbWneKTolQKFYtWE7Wv44+vLK9yXdzYvGox6pGkEU4 9i02VEJE Gos6+qkhDeRFqrlR3AJ7giahfFaMHxsUWuitY3vnpPuS0qLWXGxMGHJTbh23m90FtWvNj4bwOTc35V+xn61qc3gkL8CeyrK1ZeqkE0Eptwt7CiDOv4S2/qIpBDgluwhXsebshaIreq0Re/o4XJ+CHxh9Qa6yUg1wIbD44ob9MqeZQRFX7PfTZq+v/HBhlreSKRFDVSjWH4k0nHuOhRVl1Q71zHA4s8W1I6u++hkmYWlT7OWNHovZtCOz4AqhsctJodhoU0cUpm54NJDrQ6h8H2+s4pg+5sKf35kcbD1XgrasRTaFVGVVSbQa4PUnUN9f1Rb+VJrk2vD4ghQtu42omCUN5PRFQdWftqjlzGY8UF/lFiz8k0wJbI/5mSGXlCJH63Hn4aXUuJGY+c1/uPNnnXAenv5wfbFqhUo3o12unlatKPB/CuFCgDCK+Pk+vWcKDCQ+PmMe5wIGLVlzsKPrTMzK7gDQEDZzajDaLyeYhDntrTzLR4GyWkfowLBiFSO71dcWT4aAzxEjy8a/hbaZ1iAIGaS9x04pfBEyI2lnQ/MK7z+rO2TDgeDZzz3P1zsJBP9fkg6qJE2WxexC1pcMQXQs6JDc+KqC+EXAezpQ1389QjXk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Finish refactoring the page table logic by threading the PMC state throughout the operation, allowing us to control the operation as we go. Additionally, update the old_addr, new_addr fields in move_page_tables() as we progress through the process making use of the fact we have this state object now to track this. With these changes made, not only is the code far more readable, but we can finally transmit state throughout the entire operation, which lays the groundwork for sensibly making changes in future to how the mremap() operation is performed. Additionally take the opportunity to refactor the means of determining the progress of the operation, abstracting this to pmc_progress() and simplifying the logic to make it clearer what's going on. Signed-off-by: Lorenzo Stoakes Reviewed-by: Vlastimil Babka --- mm/internal.h | 3 + mm/mremap.c | 201 +++++++++++++++++++++++++++++--------------------- 2 files changed, 120 insertions(+), 84 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index a4608c85a3ba..b3e781855044 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -39,6 +39,9 @@ struct folio_batch; * * Use the PAGETABLE_MOVE() macro to initialise this struct. * + * The old_addr and new_addr fields are updated as the page table move is + * executed. + * * NOTE: The page table move is affected by reading from [old_addr, old_end), * and old_addr may be updated for better page table alignment, so len_in * represents the length of the range being copied as specified by the user. diff --git a/mm/mremap.c b/mm/mremap.c index a4b0124528fa..89c8d1bb9b59 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -108,8 +108,7 @@ static pmd_t *get_old_pmd(struct mm_struct *mm, unsigned long addr) return pmd; } -static pud_t *alloc_new_pud(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long addr) +static pud_t *alloc_new_pud(struct mm_struct *mm, unsigned long addr) { pgd_t *pgd; p4d_t *p4d; @@ -122,13 +121,12 @@ static pud_t *alloc_new_pud(struct mm_struct *mm, struct vm_area_struct *vma, return pud_alloc(mm, p4d, addr); } -static pmd_t *alloc_new_pmd(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long addr) +static pmd_t *alloc_new_pmd(struct mm_struct *mm, unsigned long addr) { pud_t *pud; pmd_t *pmd; - pud = alloc_new_pud(mm, vma, addr); + pud = alloc_new_pud(mm, addr); if (!pud) return NULL; @@ -172,17 +170,19 @@ static pte_t move_soft_dirty_pte(pte_t pte) return pte; } -static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, - unsigned long old_addr, unsigned long old_end, - struct vm_area_struct *new_vma, pmd_t *new_pmd, - unsigned long new_addr, bool need_rmap_locks) +static int move_ptes(struct pagetable_move_control *pmc, + unsigned long extent, pmd_t *old_pmd, pmd_t *new_pmd) { + struct vm_area_struct *vma = pmc->old; bool need_clear_uffd_wp = vma_has_uffd_without_event_remap(vma); struct mm_struct *mm = vma->vm_mm; pte_t *old_pte, *new_pte, pte; pmd_t dummy_pmdval; spinlock_t *old_ptl, *new_ptl; bool force_flush = false; + unsigned long old_addr = pmc->old_addr; + unsigned long new_addr = pmc->new_addr; + unsigned long old_end = old_addr + extent; unsigned long len = old_end - old_addr; int err = 0; @@ -204,7 +204,7 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, * serialize access to individual ptes, but only rmap traversal * order guarantees that we won't miss both the old and new ptes). */ - if (need_rmap_locks) + if (pmc->need_rmap_locks) take_rmap_locks(vma); /* @@ -278,7 +278,7 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, pte_unmap(new_pte - 1); pte_unmap_unlock(old_pte - 1, old_ptl); out: - if (need_rmap_locks) + if (pmc->need_rmap_locks) drop_rmap_locks(vma); return err; } @@ -293,10 +293,11 @@ static inline bool arch_supports_page_table_move(void) #endif #ifdef CONFIG_HAVE_MOVE_PMD -static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, - unsigned long new_addr, pmd_t *old_pmd, pmd_t *new_pmd) +static bool move_normal_pmd(struct pagetable_move_control *pmc, + pmd_t *old_pmd, pmd_t *new_pmd) { spinlock_t *old_ptl, *new_ptl; + struct vm_area_struct *vma = pmc->old; struct mm_struct *mm = vma->vm_mm; bool res = false; pmd_t pmd; @@ -342,7 +343,7 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, * We don't have to worry about the ordering of src and dst * ptlocks because exclusive mmap_lock prevents deadlock. */ - old_ptl = pmd_lock(vma->vm_mm, old_pmd); + old_ptl = pmd_lock(mm, old_pmd); new_ptl = pmd_lockptr(mm, new_pmd); if (new_ptl != old_ptl) spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); @@ -359,7 +360,7 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, VM_BUG_ON(!pmd_none(*new_pmd)); pmd_populate(mm, new_pmd, pmd_pgtable(pmd)); - flush_tlb_range(vma, old_addr, old_addr + PMD_SIZE); + flush_tlb_range(vma, pmc->old_addr, pmc->old_addr + PMD_SIZE); out_unlock: if (new_ptl != old_ptl) spin_unlock(new_ptl); @@ -368,19 +369,19 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, return res; } #else -static inline bool move_normal_pmd(struct vm_area_struct *vma, - unsigned long old_addr, unsigned long new_addr, pmd_t *old_pmd, - pmd_t *new_pmd) +static inline bool move_normal_pmd(struct pagetable_move_control *pmc, + pmd_t *old_pmd, pmd_t *new_pmd) { return false; } #endif #if CONFIG_PGTABLE_LEVELS > 2 && defined(CONFIG_HAVE_MOVE_PUD) -static bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr, - unsigned long new_addr, pud_t *old_pud, pud_t *new_pud) +static bool move_normal_pud(struct pagetable_move_control *pmc, + pud_t *old_pud, pud_t *new_pud) { spinlock_t *old_ptl, *new_ptl; + struct vm_area_struct *vma = pmc->old; struct mm_struct *mm = vma->vm_mm; pud_t pud; @@ -406,7 +407,7 @@ static bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr, * We don't have to worry about the ordering of src and dst * ptlocks because exclusive mmap_lock prevents deadlock. */ - old_ptl = pud_lock(vma->vm_mm, old_pud); + old_ptl = pud_lock(mm, old_pud); new_ptl = pud_lockptr(mm, new_pud); if (new_ptl != old_ptl) spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); @@ -418,7 +419,7 @@ static bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr, VM_BUG_ON(!pud_none(*new_pud)); pud_populate(mm, new_pud, pud_pgtable(pud)); - flush_tlb_range(vma, old_addr, old_addr + PUD_SIZE); + flush_tlb_range(vma, pmc->old_addr, pmc->old_addr + PUD_SIZE); if (new_ptl != old_ptl) spin_unlock(new_ptl); spin_unlock(old_ptl); @@ -426,19 +427,19 @@ static bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr, return true; } #else -static inline bool move_normal_pud(struct vm_area_struct *vma, - unsigned long old_addr, unsigned long new_addr, pud_t *old_pud, - pud_t *new_pud) +static inline bool move_normal_pud(struct pagetable_move_control *pmc, + pud_t *old_pud, pud_t *new_pud) { return false; } #endif #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) -static bool move_huge_pud(struct vm_area_struct *vma, unsigned long old_addr, - unsigned long new_addr, pud_t *old_pud, pud_t *new_pud) +static bool move_huge_pud(struct pagetable_move_control *pmc, + pud_t *old_pud, pud_t *new_pud) { spinlock_t *old_ptl, *new_ptl; + struct vm_area_struct *vma = pmc->old; struct mm_struct *mm = vma->vm_mm; pud_t pud; @@ -453,7 +454,7 @@ static bool move_huge_pud(struct vm_area_struct *vma, unsigned long old_addr, * We don't have to worry about the ordering of src and dst * ptlocks because exclusive mmap_lock prevents deadlock. */ - old_ptl = pud_lock(vma->vm_mm, old_pud); + old_ptl = pud_lock(mm, old_pud); new_ptl = pud_lockptr(mm, new_pud); if (new_ptl != old_ptl) spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); @@ -466,8 +467,8 @@ static bool move_huge_pud(struct vm_area_struct *vma, unsigned long old_addr, /* Set the new pud */ /* mark soft_ditry when we add pud level soft dirty support */ - set_pud_at(mm, new_addr, new_pud, pud); - flush_pud_tlb_range(vma, old_addr, old_addr + HPAGE_PUD_SIZE); + set_pud_at(mm, pmc->new_addr, new_pud, pud); + flush_pud_tlb_range(vma, pmc->old_addr, pmc->old_addr + HPAGE_PUD_SIZE); if (new_ptl != old_ptl) spin_unlock(new_ptl); spin_unlock(old_ptl); @@ -475,8 +476,9 @@ static bool move_huge_pud(struct vm_area_struct *vma, unsigned long old_addr, return true; } #else -static bool move_huge_pud(struct vm_area_struct *vma, unsigned long old_addr, - unsigned long new_addr, pud_t *old_pud, pud_t *new_pud) +static bool move_huge_pud(struct pagetable_move_control *pmc, + pud_t *old_pud, pud_t *new_pud) + { WARN_ON_ONCE(1); return false; @@ -497,10 +499,12 @@ enum pgt_entry { * destination pgt_entry. */ static __always_inline unsigned long get_extent(enum pgt_entry entry, - unsigned long old_addr, unsigned long old_end, - unsigned long new_addr) + struct pagetable_move_control *pmc) { unsigned long next, extent, mask, size; + unsigned long old_addr = pmc->old_addr; + unsigned long old_end = pmc->old_end; + unsigned long new_addr = pmc->new_addr; switch (entry) { case HPAGE_PMD: @@ -529,38 +533,54 @@ static __always_inline unsigned long get_extent(enum pgt_entry entry, return extent; } +/* + * Should move_pgt_entry() acquire the rmap locks? This is either expressed in + * the PMC, or overridden in the case of normal, larger page tables. + */ +static bool should_take_rmap_locks(struct pagetable_move_control *pmc, + enum pgt_entry entry) +{ + if (pmc->need_rmap_locks) + return true; + + switch (entry) { + case NORMAL_PMD: + case NORMAL_PUD: + return true; + default: + return false; + } +} + /* * Attempts to speedup the move by moving entry at the level corresponding to * pgt_entry. Returns true if the move was successful, else false. */ -static bool move_pgt_entry(enum pgt_entry entry, struct vm_area_struct *vma, - unsigned long old_addr, unsigned long new_addr, - void *old_entry, void *new_entry, bool need_rmap_locks) +static bool move_pgt_entry(struct pagetable_move_control *pmc, + enum pgt_entry entry, void *old_entry, void *new_entry) { bool moved = false; + bool need_rmap_locks = should_take_rmap_locks(pmc, entry); /* See comment in move_ptes() */ if (need_rmap_locks) - take_rmap_locks(vma); + take_rmap_locks(pmc->old); switch (entry) { case NORMAL_PMD: - moved = move_normal_pmd(vma, old_addr, new_addr, old_entry, - new_entry); + moved = move_normal_pmd(pmc, old_entry, new_entry); break; case NORMAL_PUD: - moved = move_normal_pud(vma, old_addr, new_addr, old_entry, - new_entry); + moved = move_normal_pud(pmc, old_entry, new_entry); break; case HPAGE_PMD: moved = IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && - move_huge_pmd(vma, old_addr, new_addr, old_entry, + move_huge_pmd(pmc->old, pmc->old_addr, pmc->new_addr, old_entry, new_entry); break; case HPAGE_PUD: moved = IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && - move_huge_pud(vma, old_addr, new_addr, old_entry, - new_entry); + move_huge_pud(pmc, old_entry, new_entry); break; default: @@ -569,7 +589,7 @@ static bool move_pgt_entry(enum pgt_entry entry, struct vm_area_struct *vma, } if (need_rmap_locks) - drop_rmap_locks(vma); + drop_rmap_locks(pmc->old); return moved; } @@ -705,19 +725,48 @@ static void try_realign_addr(struct pagetable_move_control *pmc, pmc->new_addr &= pagetable_mask; } +/* Is the page table move operation done? */ +static bool pmc_done(struct pagetable_move_control *pmc) +{ + return pmc->old_addr >= pmc->old_end; +} + +/* Advance to the next page table, offset by extent bytes. */ +static void pmc_next(struct pagetable_move_control *pmc, unsigned long extent) +{ + pmc->old_addr += extent; + pmc->new_addr += extent; +} + +/* + * Determine how many bytes in the specified input range have had their page + * tables moved so far. + */ +static unsigned long pmc_progress(struct pagetable_move_control *pmc) +{ + unsigned long orig_old_addr = pmc->old_end - pmc->len_in; + unsigned long old_addr = pmc->old_addr; + + /* + * Prevent negative return values when {old,new}_addr was realigned but + * we broke out of the loop in move_page_tables() for the first PMD + * itself. + */ + return old_addr < orig_old_addr ? 0 : old_addr - orig_old_addr; +} + unsigned long move_page_tables(struct pagetable_move_control *pmc) { - unsigned long extent, old_end; + unsigned long extent; struct mmu_notifier_range range; pmd_t *old_pmd, *new_pmd; pud_t *old_pud, *new_pud; - unsigned long old_addr, new_addr; - struct vm_area_struct *vma = pmc->old; + struct mm_struct *mm = pmc->old->vm_mm; if (!pmc->len_in) return 0; - if (is_vm_hugetlb_page(vma)) + if (is_vm_hugetlb_page(pmc->old)) return move_hugetlb_page_tables(pmc->old, pmc->new, pmc->old_addr, pmc->new_addr, pmc->len_in); @@ -726,87 +775,71 @@ unsigned long move_page_tables(struct pagetable_move_control *pmc) * Only realign if the mremap copying hits a PMD boundary. */ try_realign_addr(pmc, PMD_MASK); - /* These may have been changed. */ - old_addr = pmc->old_addr; - new_addr = pmc->new_addr; - old_end = pmc->old_end; - - flush_cache_range(vma, old_addr, old_end); - mmu_notifier_range_init(&range, MMU_NOTIFY_UNMAP, 0, vma->vm_mm, - old_addr, old_end); + + flush_cache_range(pmc->old, pmc->old_addr, pmc->old_end); + mmu_notifier_range_init(&range, MMU_NOTIFY_UNMAP, 0, mm, + pmc->old_addr, pmc->old_end); mmu_notifier_invalidate_range_start(&range); - for (; old_addr < old_end; old_addr += extent, new_addr += extent) { + for (; !pmc_done(pmc); pmc_next(pmc, extent)) { cond_resched(); /* * If extent is PUD-sized try to speed up the move by moving at the * PUD level if possible. */ - extent = get_extent(NORMAL_PUD, old_addr, old_end, new_addr); + extent = get_extent(NORMAL_PUD, pmc); - old_pud = get_old_pud(vma->vm_mm, old_addr); + old_pud = get_old_pud(mm, pmc->old_addr); if (!old_pud) continue; - new_pud = alloc_new_pud(vma->vm_mm, vma, new_addr); + new_pud = alloc_new_pud(mm, pmc->new_addr); if (!new_pud) break; if (pud_trans_huge(*old_pud) || pud_devmap(*old_pud)) { if (extent == HPAGE_PUD_SIZE) { - move_pgt_entry(HPAGE_PUD, vma, old_addr, new_addr, - old_pud, new_pud, pmc->need_rmap_locks); + move_pgt_entry(pmc, HPAGE_PUD, old_pud, new_pud); /* We ignore and continue on error? */ continue; } } else if (IS_ENABLED(CONFIG_HAVE_MOVE_PUD) && extent == PUD_SIZE) { - if (move_pgt_entry(NORMAL_PUD, vma, old_addr, new_addr, - old_pud, new_pud, true)) + if (move_pgt_entry(pmc, NORMAL_PUD, old_pud, new_pud)) continue; } - extent = get_extent(NORMAL_PMD, old_addr, old_end, new_addr); - old_pmd = get_old_pmd(vma->vm_mm, old_addr); + extent = get_extent(NORMAL_PMD, pmc); + old_pmd = get_old_pmd(mm, pmc->old_addr); if (!old_pmd) continue; - new_pmd = alloc_new_pmd(vma->vm_mm, vma, new_addr); + new_pmd = alloc_new_pmd(mm, pmc->new_addr); if (!new_pmd) break; again: if (is_swap_pmd(*old_pmd) || pmd_trans_huge(*old_pmd) || pmd_devmap(*old_pmd)) { if (extent == HPAGE_PMD_SIZE && - move_pgt_entry(HPAGE_PMD, vma, old_addr, new_addr, - old_pmd, new_pmd, pmc->need_rmap_locks)) + move_pgt_entry(pmc, HPAGE_PMD, old_pmd, new_pmd)) continue; - split_huge_pmd(vma, old_pmd, old_addr); + split_huge_pmd(pmc->old, old_pmd, pmc->old_addr); } else if (IS_ENABLED(CONFIG_HAVE_MOVE_PMD) && extent == PMD_SIZE) { /* * If the extent is PMD-sized, try to speed the move by * moving at the PMD level if possible. */ - if (move_pgt_entry(NORMAL_PMD, vma, old_addr, new_addr, - old_pmd, new_pmd, true)) + if (move_pgt_entry(pmc, NORMAL_PMD, old_pmd, new_pmd)) continue; } if (pmd_none(*old_pmd)) continue; if (pte_alloc(pmc->new->vm_mm, new_pmd)) break; - if (move_ptes(vma, old_pmd, old_addr, old_addr + extent, - pmc->new, new_pmd, new_addr, pmc->need_rmap_locks) < 0) + if (move_ptes(pmc, extent, old_pmd, new_pmd) < 0) goto again; } mmu_notifier_invalidate_range_end(&range); - /* - * Prevent negative return values when {old,new}_addr was realigned - * but we broke out of the above loop for the first PMD itself. - */ - if (old_addr < old_end - pmc->len_in) - return 0; - - return pmc->len_in + old_addr - old_end; /* how much done */ + return pmc_progress(pmc); } /* Set vrm->delta to the difference in VMA size specified by user. */