From patchwork Thu Jun 27 00:54:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 13713671 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B99C6C30653 for ; Thu, 27 Jun 2024 00:58:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 49B786B00AF; Wed, 26 Jun 2024 20:58:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 44BB16B00B0; Wed, 26 Jun 2024 20:58:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 277CA6B00B1; Wed, 26 Jun 2024 20:58:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 034306B00AF for ; Wed, 26 Jun 2024 20:58:28 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9F2BF16171A for ; Thu, 27 Jun 2024 00:58:28 +0000 (UTC) X-FDA: 82274857896.15.6230BCA Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2053.outbound.protection.outlook.com [40.107.220.53]) by imf24.hostedemail.com (Postfix) with ESMTP id AB02518000E for ; Thu, 27 Jun 2024 00:58:25 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=kOUHuXR2; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf24.hostedemail.com: domain of apopple@nvidia.com designates 40.107.220.53 as permitted sender) smtp.mailfrom=apopple@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719449889; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rl2/HlMeBklMNcqOSvoLzCBY1HST5wM2+mfyJFbUiaU=; b=7DB6h2LNYlTImt/cHRiA76trrg0F1T/oQGq3JBBeWxHL1SlkIaGGafaDsWqc7jmls36MgY D4nYIJEk/BNccvnZi0oe1nmFognOKjECWZKHRLUoMCRkv5Jpu4iMIgBGBWC6zes3twFHro xLWt9tbd0Q7pOdZHt9lHIevwn7FgFuc= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1719449889; a=rsa-sha256; cv=pass; b=aUsJiJzZSR2DQ2hO6ylYnnmyRDz86njs22oMgHHFjqqGyc8K5GXOET4AsiK1Pl5eLjV4fD 4wk8AMUVKXi58rY7ld2Wmz24/bY/3WaFbeBqGAMNVHWkbyQzBMrxFdH8pK/Polt9j6jolc Od98tblInCwxXTuyXFZ9O4umtDA0fSE= ARC-Authentication-Results: i=2; imf24.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=kOUHuXR2; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf24.hostedemail.com: domain of apopple@nvidia.com designates 40.107.220.53 as permitted sender) smtp.mailfrom=apopple@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Y/+1im9Wg+1Hj8R5eBNK8Ghn4Zm1iAhxp9RbURrEcsMP4+RBF1l40BlsOmF395e4rz9r30Dun4Ng1PJTr3NQZkUxIoyVojbDuheWHDYiisBuJbfsXwEA6sp3cgSvH4wPzCnU0XGg7a02TY2uWFJIFkTk2pEiXA0/14br45vP98u23N4b49VTOnVR94/vH7au0kpmrvxX1I6uacq0lyDXwiHU59sRbSttJihZiXYcLwEUfd8GlOfxtQVrotxNEvHewG/pTxf9npSgXBEFEoYvrMWk3e0PQCLbDiajnkG94ESUvo5n4vtfRDpdg0mNfq81gtgnWau9Zs0JiAFVYzMiCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rl2/HlMeBklMNcqOSvoLzCBY1HST5wM2+mfyJFbUiaU=; b=VYupiNs2b3HFY7Emjyqi51Bus0msopOJQT4/7fLXaaLQoghV4A3FDqV/7NRg8IpVt6LdvuKDuFSlNcT41hamZX13MW2eVDPbLEa3PlTU2hQpQSxENF4Ic5TOeQbX9s7Z6hYU0ldzY2Q2/7JosGIJzq37+yqEnB5tQP60quzCCoJeobOjDsNorkqtYJaj2kj0WvpDwDzBoBKWsPJHI4xrBbxsUykKxY8xUUBWVWg6201yNsNNjwXh7T2ypJaxmGaU0rvxcobcCurborophAR5vx9XskHt6J7L9fFQND0GV8nGUpQZV2EPEd3C7KFS6Gx8dntFJRaltVBnWkzmLdZgRA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rl2/HlMeBklMNcqOSvoLzCBY1HST5wM2+mfyJFbUiaU=; b=kOUHuXR21vB7ce/V7GuZoaUnCR6Ly3cOMrMN5O+55V5E2ePUCoeYQZHtU9al7B39p8nBcnKZYejwz8vIXQVjpxUfLS6gS/WJFx6Vz4s0XNX0WTuRdPQw1ugoiRCDFV2UChqWe1PbFd4b7I+jcGfr++XMLnbpCb5o5t2EqTbWagIF5dC1FmZM56zTkIryIASqfF73MlVClejuyWK6PLPv6Z/5ge5YSjlAyI8R7wQsvXmofaWax0noBOWKeNfM3OhfNCzs/5y1gCMbXECrABa77E/hrrt+LQGfsAY2rvsOYmSOQYieMikdjEOcwzLw2JNeeonfL6tqPlLuUn8C5rpBYw== Received: from DS0PR12MB7726.namprd12.prod.outlook.com (2603:10b6:8:130::6) by MW6PR12MB7071.namprd12.prod.outlook.com (2603:10b6:303:238::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7698.32; Thu, 27 Jun 2024 00:54:58 +0000 Received: from DS0PR12MB7726.namprd12.prod.outlook.com ([fe80::953f:2f80:90c5:67fe]) by DS0PR12MB7726.namprd12.prod.outlook.com ([fe80::953f:2f80:90c5:67fe%6]) with mapi id 15.20.7698.025; Thu, 27 Jun 2024 00:54:58 +0000 From: Alistair Popple To: dan.j.williams@intel.com, vishal.l.verma@intel.com, dave.jiang@intel.com, logang@deltatee.com, bhelgaas@google.com, jack@suse.cz, jgg@ziepe.ca Cc: catalin.marinas@arm.com, will@kernel.org, mpe@ellerman.id.au, npiggin@gmail.com, dave.hansen@linux.intel.com, ira.weiny@intel.com, willy@infradead.org, djwong@kernel.org, tytso@mit.edu, linmiaohe@huawei.com, david@redhat.com, peterx@redhat.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, nvdimm@lists.linux.dev, linux-cxl@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, jhubbard@nvidia.com, hch@lst.de, david@fromorbit.com, Alistair Popple , Jason Gunthorpe Subject: [PATCH 05/13] mm: Allow compound zone device pages Date: Thu, 27 Jun 2024 10:54:20 +1000 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: X-ClientProxiedBy: SY6PR01CA0046.ausprd01.prod.outlook.com (2603:10c6:10:e9::15) To DS0PR12MB7726.namprd12.prod.outlook.com (2603:10b6:8:130::6) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR12MB7726:EE_|MW6PR12MB7071:EE_ X-MS-Office365-Filtering-Correlation-Id: f711c151-0360-4be8-10a1-08dc9643c443 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: I+Ji6DziS2gkSBQKiQfxM5SJwLCreKhG+0QwMpZvAC2RABGzBiRnU7q4c8BvHw/Ksx4GXTqSvm99bfrxKMNCYaBq1D3WGBQMJ61FHarpVDilQKFMZ0ilzGV++PwujpBoS2Fvkp0eWMkaITtHavAC9/87++syCxczWfG2IzX7RRbPKu3riLUy18QOO6FnUhAepKuZrHyvavlx7cDd84v6NTzy1ryPUnCiU9Ern/yEbt4jppIbQhiWYE9dH6yt7+o8dPXHC6i7z5s0XpWRFrT5+9l/d4cGJ6GPid0owrxMaqbCufTQ1Cp7hbZCSxkvlBvrLPEMVXsfd9iuIghdFE+eo9n3B+NMqca8q8eH3Z2iWgEaG8/sGIFIj52ttpvn4hVXdTDJMPQktthkd1C07jLD2otWReGHB/xemh7ifiTfjoACXyTuLPeau2YceaCgwXwPWBw9FODDxGEsZ+3rt+79ZbJYpadGBtFzqNY4T8NwpmPQ78Sivor21NkmphqGp1G97jJqgzbGYbh2wEZSt3lGp/Y8zE5E0IS5cqk9eg6isHJp1caalP/te6uskAa/g0At2Z1ryoOKwv0YNkWqqA8QqYkTyfE6P9tSYfb+jf/gGFnqj/7wviS0Wui3YOLxDlDXuSJaHJe4sEnAq1vUW1908kVoc9u27Tqy4wCVGfhMGD3dHGq+BD5BRFXIZB9jtf3J29Gfc2jDeA9ouRvsZvck5dUdmRpCwfAq8MGZ04/2Beu4DbltfkxInksKgzrcHvoV+pkBfLqSR1iSHv3WW6hp8py3ZJpWOOtF6YFNWipLhRoY4c+laS6NE4ZH7lM4JXo2VxcWWXQZggd0DjQxlqI1/TxP/WFj+x1pJD7kKX84H6nkXGTmlJLDBr4iHRwhzeaqpCud2wx3nyKwzYd+LbIqgjBAReMwr6SmGNuqQOr77nJD2YQftm6Pa+uZU/vJ7SOZ1MUa30SSmCdYzKksB7iEGmuGD7vUWUed0j3nC2kGiTRqcOJ8o5jgmBsffr/j3hzX3kOP8pdq7rJNBXVRKVX2sndFQ0BZb3tCT68gyYT9J/xXMP81gYdP75ZEFr+OREijch5jxLgzVWXYQI1n1r2n4veyde8jH2JvlXl69zsP7lVeVZCDnoH30FRX1UU/cUvRYwHCu0fWNq9bwBLVvIQDqCeV2Snim/AlIebjsQuR2Edsyp81U2ZMRvGkVh6cnKwr7BIyUlBCdm3a48zkagJmwKdNmbqv9qrouqV4dzsCqFNyy92ECKr0s4wIOHik4fpH19k/Nc9Jks6l9LUDTeg2tipSPyMt4Wor2D73wJxPmzX2vmY4VdvvJ6SWO4jZULPGqZRv7FQX3KCuTUJSIKtjCQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS0PR12MB7726.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: eg8znclncNP70gE2sr1mpDccRz5CdT4JR/6cDRuJxQfgnqNkDBCyE/HCzZgw/D8xQEEBxmQkrwnGWiSGbcHGkP49lPpLn4zw7+U4hn7aGs8RINWsie1kd468qNBXmt4IulTYb176mDV8vbYayQJ6OSULS6wEBYqDCrOHSr+rrsnh0uKFk2JATIVOBJwbNsIRrXReLfb0hyZ/UMSUGXAAKYnr/4EOTApxnzmLsPpxhYBuNtBiKUj9oHCzMzHY6CK8Iq1wXmGV5GrZUzw1sRQSrTWuD8xo57FOFKfVJzvqqRuIKjFKppP5htNPbAQ61q0fudAWdqTxFVjOlO/Znfc9ucHNmjEg4clzGl4Ecr9GgXYB0Q7wZqtiHWkURhnl92M/GTm6Wie9HLg7zQmgzVijUKfjw9s//oJzL/DS9q00GGbmHZpvjOwNcx+K7OedcuyDBiqg/zlxRC89rG/+kRRy2ZXYQLuGT3XlTxmqDBQKX7N/M6I1DmOyqdS7rU9WZFODjxfezqQsPa+ODATdIdSvxHm73z3Ja5Ctc49Zz8MpqphLKIhtACoFZK4kDpaoj0kOvjFObWVZEB7BYkP5huWrInZ1kaiMTjKhhjA+Yvk+V/gOXjr1T9Nt48gh8bOrl6Bn5kEguB3IDMIVn2i3FYbSgt7ebSV87VEEMsPs4ZhbY4GTarxWRVCUSeje7ZoBuhcWhK2Sngha0D+s5M7OpvfsHwuoPJZX2Z7JUqnd+hlZ6wc+rlJye+lOdMAHwix8W1AjWWeA01fkijw+XaPgHLrXmsI2cPyExA1Kt2z8BO0kIiDC5lQlA7zy98bjw/HbwjCWCMvAToKxxpYXEntzf0GaiMGpMz3tsLSEnGZuRmcVvMNA4RpGervMh7EjhJsF5JpwcINNwkp2cRoHSLJRFLqyFqhltaoLG8NE9Mfvi2AmOxBwkjX/kGkJgu7GN5QGeBaSxVOfQTIH1SYNdXf08p2w3kwcxQdss9cOMMZSpbty5wtSWfZUGPSLpJ4Vp1sno0H7exCI4D8GpILYrbemqdY+ByLRgUPj2X7sr/NYFsti86/eZzHSr22tre+HtJsACJSMpqxet/+DHa/O4x4HS+ZRweY3U4nwYhSZSq9JVTqmisMmqmQc/2vxyIptj9OfEaOccG9FCgkK3wgucqbIP4nTHcRu7lNwytH1fSNnZkSZrXA8sdrh/etJT5/yiyAHb/INvjcTckpJqnCS1pszbZUdazWaZLFB9nd8C2GbY99ibH5KbFnIuAK5h6UV6RoD3uPHFiqz/TSRlOfkmuUMkJHS8d9pmOCxLPFgikFYBT0jrshK6KHxvbx/mSeADY6qphgrI7lYDvAZR/UbnKfVQay4Sz2uFJVRXSgD30OTQnoP+zd1alTBQRIS+0oPXba7Cd/yQdi59e+w5kIUlvc1U4RVW+r4iMJFcEi7khu/c+F0QuAeWzuz7t8CZ4HVa31fiJCwMdK/DuIEBzesKcPv5qY9hClXlQWHulo4UjYw6kTS14syFEJO/XFQbFQYJj1HJD59o/fAZacTGthbxKKyiHYL45P02JDtOVt9/N5fIgNSV8g7xOeVG6Mck64Ye7V3lJbV X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: f711c151-0360-4be8-10a1-08dc9643c443 X-MS-Exchange-CrossTenant-AuthSource: DS0PR12MB7726.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jun 2024 00:54:58.1109 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: tbwm5sX7q/04XauDJ1H5ddMRfz6qo7yI/87yFCY3o44qcGCzXLX2dDCYOwcdXdzP4mFGU1u0mvsUFn6L9nbGQg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW6PR12MB7071 X-Stat-Signature: uj8msisw5n51hidfo4s31kjcx3dpbmki X-Rspamd-Queue-Id: AB02518000E X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1719449905-207881 X-HE-Meta: U2FsdGVkX18mRH8MAO9FUzaZvQQfjsoLcIjj4mWuUDaeMtcg1atbiJcD6048VsJU8dwuuRjsTlYzlmAs5qmLts0Rmu6HxfeT1oMo1Vbp87R9OUS/ya/t+PFWUeAsAge+eZo47Q6IJcXe9vwoEtLu2B48LQk5VvdzJLda51M1jTr7qUU8La7sMcZZ6fI/8SfuE+MfsHTB4kyrHAEnEnua479cye0aEsb/NG52fgtbaVwRkyOGqz2BK2oj1G3OEgqvrhI2E6Q2HeJ2Yffgj7rxNAvmzhAN/fSuvG7DQ26TQzHXmwmOuiMXUMbCLQ7AYna6Pc4Y00ti8hFrTDIvgyGD38NnE2ZQGpDxdxxkEg963VS6CqKwMnFPSq9w6UGhQ4tAddU2EGoE4n9FnbTN/fCe1bR4hgpreBSidONKl6QojgNM4nRGoIvjMAWaLNbTW6yO2EkrEh8CsmAh9qVQbBcCL9z1FaV/c7RS4ixcW2V7Vb1SAkCmv9fV95e65xOnnfxmo2KCwz+La0IVTD0JDZATbml6jslubmBn9RTAbYFsGNOupG7v01yeMQtR0bq2xk3bRHolsL0sSkMpaWaOf2QDUA70WG5r1DuQNDAQidD35FWwXVwA5/tLXmZ8uthvohQhu8f6KM9rMkxZ7VsPGXlXqmFcMjwZ5z8pwFgtSJZHNtWuxBz2YjzJEBqDaUp0Su5APqpZHZp+ksKsknJsAO4tMuohYGvwuKs73TUfH4qqyHcp/bE3gchDyS6vx1nJ9UNAsAR9M6y6i69k3+QrzidNUzR1FQQj5Dv52cyx56Jn8dF1l6049u1Xd/R3UCni8GVPMF+zoG8vNzmHs1YKomJD/Z80tIizcTRzOlQ47Ae3yIsj9J1qh0/IJFt5hrWAMr8ARTfUgU14K7B0ZpXxNm4ZdXunWoDiftb7xMoT+sooMWfWYuPU9X2HsauTfufk3XXNmuJnkGJvPdGBV//VDSI oAgPPX2r 1kQtHCf/1bMMMmEN49Bedx6ss8xZXDAQzq2kWwUjtyoqRnic9pJ4ZLvv0nmepXMJyrw5AuROMykw/JGnZ6Rp98XGUrgr7xofVp4GoXkRqYLsic4rDwjPbJJt+TnveptA0XTYdRXeRyjb9eZaD+x3b6Ddbdqk87+idmthX/Xvs69yvbjeDaQ9jO1A5B7rKo+jr3KDgY/DqZEwouWgZNH6r5Y2SatMUOPIroWNmuJmZtRKF+gpCk+0nPRctSGmLmY60EZTBQcm4Esh5djs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Zone device pages are used to represent various type of device memory managed by device drivers. Currently compound zone device pages are not supported. This is because MEMORY_DEVICE_FS_DAX pages are the only user of higher order zone device pages and have their own page reference counting. A future change will unify FS DAX reference counting with normal page reference counting rules and remove the special FS DAX reference counting. Supporting that requires compound zone device pages. Supporting compound zone device pages requires compound_head() to distinguish between head and tail pages whilst still preserving the special struct page fields that are specific to zone device pages. A tail page is distinguished by having bit zero being set in page->compound_head, with the remaining bits pointing to the head page. For zone device pages page->compound_head is shared with page->pgmap. The page->pgmap field is common to all pages within a memory section. Therefore pgmap is the same for both head and tail pages and we can use the same scheme to distinguish tail pages. To obtain the pgmap for a tail page a new accessor is introduced to fetch it from compound_head. Signed-off-by: Alistair Popple Reviewed-by: Jason Gunthorpe --- In response to the RFC Matthew Wilcox pointed out that we could move the pgmap field to the folio. Morally I think that's where pgmap belongs, so I it's a good idea that I just haven't had a change to implement yet. I suspect there will be at least a v2 of this series though so will probably do it then. --- drivers/gpu/drm/nouveau/nouveau_dmem.c | 2 +- drivers/pci/p2pdma.c | 2 +- include/linux/memremap.h | 12 +++++++++--- include/linux/migrate.h | 2 +- lib/test_hmm.c | 2 +- mm/hmm.c | 2 +- mm/memory.c | 2 +- mm/memremap.c | 8 ++++---- mm/migrate_device.c | 4 ++-- 9 files changed, 21 insertions(+), 15 deletions(-) diff --git a/drivers/gpu/drm/nouveau/nouveau_dmem.c b/drivers/gpu/drm/nouveau/nouveau_dmem.c index 6fb65b0..18d74a7 100644 --- a/drivers/gpu/drm/nouveau/nouveau_dmem.c +++ b/drivers/gpu/drm/nouveau/nouveau_dmem.c @@ -88,7 +88,7 @@ struct nouveau_dmem { static struct nouveau_dmem_chunk *nouveau_page_to_chunk(struct page *page) { - return container_of(page->pgmap, struct nouveau_dmem_chunk, pagemap); + return container_of(page_dev_pagemap(page), struct nouveau_dmem_chunk, pagemap); } static struct nouveau_drm *page_to_drm(struct page *page) diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 1e9ea32..d9b422a 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -195,7 +195,7 @@ static const struct attribute_group p2pmem_group = { static void p2pdma_page_free(struct page *page) { - struct pci_p2pdma_pagemap *pgmap = to_p2p_pgmap(page->pgmap); + struct pci_p2pdma_pagemap *pgmap = to_p2p_pgmap(page_dev_pagemap(page)); /* safe to dereference while a reference is held to the percpu ref */ struct pci_p2pdma *p2pdma = rcu_dereference_protected(pgmap->provider->p2pdma, 1); diff --git a/include/linux/memremap.h b/include/linux/memremap.h index 3f7143a..6505713 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -140,6 +140,12 @@ struct dev_pagemap { }; }; +static inline struct dev_pagemap *page_dev_pagemap(const struct page *page) +{ + WARN_ON(!is_zone_device_page(page)); + return compound_head(page)->pgmap; +} + static inline bool pgmap_has_memory_failure(struct dev_pagemap *pgmap) { return pgmap->ops && pgmap->ops->memory_failure; @@ -161,7 +167,7 @@ static inline bool is_device_private_page(const struct page *page) { return IS_ENABLED(CONFIG_DEVICE_PRIVATE) && is_zone_device_page(page) && - page->pgmap->type == MEMORY_DEVICE_PRIVATE; + page_dev_pagemap(page)->type == MEMORY_DEVICE_PRIVATE; } static inline bool folio_is_device_private(const struct folio *folio) @@ -173,13 +179,13 @@ static inline bool is_pci_p2pdma_page(const struct page *page) { return IS_ENABLED(CONFIG_PCI_P2PDMA) && is_zone_device_page(page) && - page->pgmap->type == MEMORY_DEVICE_PCI_P2PDMA; + page_dev_pagemap(page)->type == MEMORY_DEVICE_PCI_P2PDMA; } static inline bool is_device_coherent_page(const struct page *page) { return is_zone_device_page(page) && - page->pgmap->type == MEMORY_DEVICE_COHERENT; + page_dev_pagemap(page)->type == MEMORY_DEVICE_COHERENT; } static inline bool folio_is_device_coherent(const struct folio *folio) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 2ce13e8..e31acc0 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -200,7 +200,7 @@ struct migrate_vma { unsigned long end; /* - * Set to the owner value also stored in page->pgmap->owner for + * Set to the owner value also stored in page_dev_pagemap(page)->owner for * migrating out of device private memory. The flags also need to * be set to MIGRATE_VMA_SELECT_DEVICE_PRIVATE. * The caller should always set this field when using mmu notifier diff --git a/lib/test_hmm.c b/lib/test_hmm.c index b823ba7..a02d709 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -195,7 +195,7 @@ static int dmirror_fops_release(struct inode *inode, struct file *filp) static struct dmirror_chunk *dmirror_page_to_chunk(struct page *page) { - return container_of(page->pgmap, struct dmirror_chunk, pagemap); + return container_of(page_dev_pagemap(page), struct dmirror_chunk, pagemap); } static struct dmirror_device *dmirror_page_to_device(struct page *page) diff --git a/mm/hmm.c b/mm/hmm.c index 93aebd9..26e1905 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -248,7 +248,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, * just report the PFN. */ if (is_device_private_entry(entry) && - pfn_swap_entry_to_page(entry)->pgmap->owner == + page_dev_pagemap(pfn_swap_entry_to_page(entry))->owner == range->dev_private_owner) { cpu_flags = HMM_PFN_VALID; if (is_writable_device_private_entry(entry)) diff --git a/mm/memory.c b/mm/memory.c index 25a77c4..ce48a05 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3994,7 +3994,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ get_page(vmf->page); pte_unmap_unlock(vmf->pte, vmf->ptl); - ret = vmf->page->pgmap->ops->migrate_to_ram(vmf); + ret = page_dev_pagemap(vmf->page)->ops->migrate_to_ram(vmf); put_page(vmf->page); } else if (is_hwpoison_entry(entry)) { ret = VM_FAULT_HWPOISON; diff --git a/mm/memremap.c b/mm/memremap.c index caccbd8..13c1d5b 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -458,8 +458,8 @@ EXPORT_SYMBOL_GPL(get_dev_pagemap); void free_zone_device_folio(struct folio *folio) { - if (WARN_ON_ONCE(!folio->page.pgmap->ops || - !folio->page.pgmap->ops->page_free)) + if (WARN_ON_ONCE(!page_dev_pagemap(&folio->page)->ops || + !page_dev_pagemap(&folio->page)->ops->page_free)) return; mem_cgroup_uncharge(folio); @@ -486,7 +486,7 @@ void free_zone_device_folio(struct folio *folio) * to clear folio->mapping. */ folio->mapping = NULL; - folio->page.pgmap->ops->page_free(folio_page(folio, 0)); + page_dev_pagemap(&folio->page)->ops->page_free(folio_page(folio, 0)); if (folio->page.pgmap->type == MEMORY_DEVICE_PRIVATE || folio->page.pgmap->type == MEMORY_DEVICE_COHERENT) @@ -505,7 +505,7 @@ void zone_device_page_init(struct page *page) * Drivers shouldn't be allocating pages after calling * memunmap_pages(). */ - WARN_ON_ONCE(!percpu_ref_tryget_live(&page->pgmap->ref)); + WARN_ON_ONCE(!percpu_ref_tryget_live(&page_dev_pagemap(page)->ref)); set_page_count(page, 1); lock_page(page); } diff --git a/mm/migrate_device.c b/mm/migrate_device.c index aecc719..4fdd8fa 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -135,7 +135,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, page = pfn_swap_entry_to_page(entry); if (!(migrate->flags & MIGRATE_VMA_SELECT_DEVICE_PRIVATE) || - page->pgmap->owner != migrate->pgmap_owner) + page_dev_pagemap(page)->owner != migrate->pgmap_owner) goto next; mpfn = migrate_pfn(page_to_pfn(page)) | @@ -156,7 +156,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, goto next; else if (page && is_device_coherent_page(page) && (!(migrate->flags & MIGRATE_VMA_SELECT_DEVICE_COHERENT) || - page->pgmap->owner != migrate->pgmap_owner)) + page_dev_pagemap(page)->owner != migrate->pgmap_owner)) goto next; mpfn = migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE; mpfn |= pte_write(pte) ? MIGRATE_PFN_WRITE : 0;