From patchwork Thu Aug 15 15:11:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13764896 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 078E8C3DA7F for ; Thu, 15 Aug 2024 15:11:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 847846B0123; Thu, 15 Aug 2024 11:11:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7F7E76B0124; Thu, 15 Aug 2024 11:11:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6710D6B0125; Thu, 15 Aug 2024 11:11:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 455B56B0123 for ; Thu, 15 Aug 2024 11:11:46 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D05A01C4FFE for ; Thu, 15 Aug 2024 15:11:45 +0000 (UTC) X-FDA: 82454819370.12.0127A15 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2074.outbound.protection.outlook.com [40.107.92.74]) by imf30.hostedemail.com (Postfix) with ESMTP id 106E180021 for ; Thu, 15 Aug 2024 15:11:42 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=PDDTmYK7; spf=pass (imf30.hostedemail.com: domain of jgg@nvidia.com designates 40.107.92.74 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1723734666; a=rsa-sha256; cv=pass; b=LrCbVHD498WBnLFNvCS+IWxmIqh4ryS6Vxv8gvcOvOPogZe5Buh/Qkah3Cnk+QW/6MzOW8 cW6B97otbZdOm5iGGigx3iCN8f5A0Xaz8CXizQ7QO5fufiNHCRW7Hvfc6gV6iLSv1QVEqz HRaBcR6f+8htfRluEEQchoTV8b/Mo/8= ARC-Authentication-Results: i=2; imf30.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=PDDTmYK7; spf=pass (imf30.hostedemail.com: domain of jgg@nvidia.com designates 40.107.92.74 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723734666; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cT5OYTyoXYx5X/x38jJiW1W79T0WeY2Tg2D25rF5DZI=; b=wO7jEvDMcd7AGW9bxzCInBVDQoPtY0XM1fs/odbnLWgTx4gRHmVtcAM2b8qVf83APYzRw/ qqdUeFxqQmeweSxv1oj56452k3qZUMiOGXJqbyYIttJwLxA2ks/4Ww+2uGEl+3FloNa8Bg zj+fA1AV8kWIqren8mo50t2b/eGB2+g= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=aHaKy78KbNTc0SAtOlDhX/zR6XSuD8DyymaN6LTXhHksRIuDR/iWAgUgo9MXfpr5zBfi9VcIxyKB+L8KhxXRWNUGvsJchnrdc7eP7PIeYNjvMPYm3POq0GaUE4Bc6+4i9hd+IEvM9f0Dbj9wQWOci191fTHk2DHaDTW6M37Hoh/tdgzRZhVeB1bZcQXjBZAK3gS7xFbHZK1aOdeq1Pzg8BX/ya43JnhXQtIc8Lu1ysYh8iWngrsAtcAsOBPXAHJxIN8KZQwVPIV+5qmngXZNoMyNwHiWFSyyKVdws3Xfpy68SPLar/AftEBsPOUkySPAe+3DJrXWoO0eGjDkVmP7jA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cT5OYTyoXYx5X/x38jJiW1W79T0WeY2Tg2D25rF5DZI=; b=gqSbBNBZ0qLxXxCPhNZUyeEwcpafirGWgcCYpjMYodStmRHt/LEQ0titBlHfFjtMqa2yV7H/9vZqDiWBwHiwBpYk25p9wiUl8JeYk8dsuvKa5gYmh0Bc8O+bYgD4iGt0r2650RWZNYzemn+e9BxRCMbUyoYlOUmGfIp6rHlLn/BS9xTrDYmdzMvG5R9X04fqck7HSbxTK3xfoXLJKOA5HDAo3YBz3aJo6/K9SZYd7Sq6eKNh++iM15vqtd6aLT1Fj5kBXrVyigUtnvhxTi/t0zJwQctYV9qCjFHDzbdrBgbdbwshCJtXNZVqruBrWV1y80rkbcesu4KPK+aV1eIxnA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cT5OYTyoXYx5X/x38jJiW1W79T0WeY2Tg2D25rF5DZI=; b=PDDTmYK7316KTvAwV7FLzV8k3mZmXVLDVlofUDnukLePag6A2/aXe4cPJcU8SP23eCGV/y/CEBtPl4xszH6YY6q1nB89vXoDDkjInlETMtRRnuroM/glx0t5ZTP/JOQjDseOrxH1BJ1TsAaWBFPOJhZL6eafgKQyF0pAYsYbTUzdT0EatlE2XjNe/DzYegcyDhK8p4ifwJysfBaGhnuylETndQKqqTQfKBH8S71/LCJOEhICa+ogw97QdOGRbqkYMTSvXUfGieK6oCXSFeZnzVApV5P9Y6Lqvch3fjfgsr/KKsT8KbwDMgbBbijFHf8mlm2GuuEJFfDOkb8qi2q3tA== Received: from CH3PR12MB7763.namprd12.prod.outlook.com (2603:10b6:610:145::10) by SN7PR12MB8146.namprd12.prod.outlook.com (2603:10b6:806:323::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7875.17; Thu, 15 Aug 2024 15:11:35 +0000 Received: from CH3PR12MB7763.namprd12.prod.outlook.com ([fe80::8b63:dd80:c182:4ce8]) by CH3PR12MB7763.namprd12.prod.outlook.com ([fe80::8b63:dd80:c182:4ce8%3]) with mapi id 15.20.7875.016; Thu, 15 Aug 2024 15:11:35 +0000 From: Jason Gunthorpe To: Cc: Alejandro Jimenez , Lu Baolu , David Hildenbrand , Christoph Hellwig , iommu@lists.linux.dev, Joao Martins , Kevin Tian , kvm@vger.kernel.org, linux-mm@kvack.org, Pasha Tatashin , Peter Xu , Ryan Roberts , Sean Christopherson , Tina Zhang Subject: [PATCH 08/16] iommupt: Add read_and_clear_dirty op Date: Thu, 15 Aug 2024 12:11:24 -0300 Message-ID: <8-v1-01fa10580981+1d-iommu_pt_jgg@nvidia.com> In-Reply-To: <0-v1-01fa10580981+1d-iommu_pt_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR05CA0049.namprd05.prod.outlook.com (2603:10b6:208:236::18) To CH3PR12MB7763.namprd12.prod.outlook.com (2603:10b6:610:145::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR12MB7763:EE_|SN7PR12MB8146:EE_ X-MS-Office365-Filtering-Correlation-Id: 6e4e0405-7df8-4d98-055f-08dcbd3c8d29 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|7416014|376014; X-Microsoft-Antispam-Message-Info: pos61fFPSFsNDtfDkVWGj5nL47JRth8Vp8W1ggD6VTclyqXVb+KbWbu5eRtkmhI4kqzX7aSkcGurLR6l9cWwbvNXNMZZO8624PunO4uta0N2zt+1MtoY09EY+dfQpNqtBrqlAw8Dgs4PoXSSdwJJJffB4Iiivc0uJ0O7zfeExnggrCBsccO029EHELZhYO1qkfUB0nL2u7ABt3uv139U1g+f6ex7/I6NAD92yN70uCG2v+wumurXM3GoHkRoqZpcPa6GmdUc6WZ5mshCz1bjzrRsplpnCN7pi9EX3G0+gFp46V0wkSXVw5yiJdJSdasZxewdPk6BbN0crlzpS+PtdrXfXoo1sqi0VIikehBBJ7uQc5eKVuXPzX6AurpQBtPDFJzj0xAgDNOND0N89UMBCABCEtPJDM7iP8TUIp4K56VS5uSfY6Dl9bW+jBNWeRaU1u748Emx0wUSBhFYqwPH98fOL6vmp1t+EuQsH4xdvq8yf1duJr5yiUC27HcVVbmP9LKQxQxMnuxAAYUWw3bL19nFKLFl0e6zNQrvIz9FCBVxFYmbV5rM8SD2pfa6cMoTufh5DtnMCLzdwsquAI0e5kvxtVTEQFi0f/uIKkZoIAnb6ImaNfwUyMcUduPBwYsKIFq+jXAy1JDI8Mz2XqnLtrGJIaPv7zfFuStplyTtGDXKaPs5D+E9sx74sg+87H+0GIp8A6jnXzsnOj6628EuAeRwMTkoNY1Pf/H/FbLo+UIMLJLy1KdCG5UtiNE6RRpar/yqa53sbt7jk99DpslRP88Tf+kcDCStaxFmlwBu5qHYr5pMwPbwhCbNV59qz2It1i8hR4d03gVvog5KtJAwCDH964Etwv/m5ow9vzwR31byG9ATvDRVbDkyc+rZwDx3iIIsn18z2mdYyF7xvLk63Wj7xIXyyjwWedB4hgPYHSq50ViTArvLbTjPxR8D/hz9pI+aCd3AbGbFCZXH0HhR7XKCrxW5ErbCmZVUzGTe5DYPSkIzWUVsC5EzrEyaKcNx+ZBmO8V65eXTITOcv1l8JByuECl6QT5Iym8Xy9VCYU51CP+Mgc5TxyLbGAlp79Lj5aSVbwyGifZMttxUjC8Fkkf7EY3D+loSWEahHWl5UhfIlGXlD0Paqeoehbu3f3zo2R7ohgzcI3T+uE2z7hfsJU5/1+BV09okcv7zJNa7IUParFOr1Y4fZLRY1O+3D5JUmkCyZ64w9q6/reo9nTumTx9wU+DRkwvra/jHlfJzIdvdpL3QJeBwYOYICnIVhcrY3KpKtVFoR5rVXOnsKNf/3/iNqAZe2QAnRl087je/3DHMnvHUHOfIf0/O6eMi6b8tkyTu8FRFZvJM5avTtmn+cQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR12MB7763.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(7416014)(376014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Ir7ZpkrRUOqHNZea6VR2AyfWPQHG1IwReI25VUlwWUB0fD/XlSQS7UmLhTzSCpocQeysA6cn9fQ3KnJgrHHMNfu05EdxRI5uspOOkLzkoGwG+zbaW6dnCprQzhtdjOxicIQo3u08QnXHgfK4FtteF7vD6uY/134c5Isq0RQjs1p8IWfwOkzbKHfQzaoRw4d2717zXYDIXKGQKEXjIqHXUactvrAF3zDihyoGtU/rTabSPd0uFOfolXZTVK5RhRB/mRDGupqy4OVXBcs5ORftOwgvQtryPxlHhv0ObnWhfqqplLkWHcKojrI1S/nXe0JeBLi2dY0GIMSDO4VrDdGv2dvmIez0BABxccrhHV/sokbm4MbTvC7SYOK62PjF1eXbM4wyLkCSL/M8gQVdbbAWOhY0Kg2IMvKCQS+uiWVwYG6KDacrRZr+WYvtH/xi2/34AthRw1JySsBp10dtlSA+dGsdn4eAdahopvsO3C4G3JssSpMqnFJA85mxN1JMtBQ0LHOGbZnb/mW0oWgJxBV8jnRxEdRkCq4GSjmVn02mRjoGUxP27/5Tq2oEeSxw60W0KRw4Cn0Nof7sxnEI3Q7BEGlQK93qhsvlBD7Wshu/CUbcXChAn50pMgUL4FCCstEsMSa1IX+ZQcVqF0wFBX7rkWKTwCjdUWSlwiHNmCq/NdszcnjHdghWQ3C2OvAJlRjVFKEaAgrFnBwC6QhYxHH5iWjIWGDEO1ESg6iIxPF+7HNrdSXkljQrI9YJV+1mMfSNEwZ/hZr7Vq39l98CWPfqbgeadVxhhrNHN5rROpABD36qO9PZVR+uhmBL8KyFWhdj3RlhhUS3rwRBvO4jWsqrVrlxgIjSKo6lOLGnoUQespTLXjOIGd4CjJe2EixDG55eRj9dAJm/T3waMPZI2KvEdZ7L9VdhEmnl9SoX7id00FIH/tf0l8NgMqHil1d11LpDq2yMh4r5pQeJ3eRUlVhuAOvsZI1ErBYXzmOxl+5cSUGrTbU7KIivsiBtoHf1bysIPsjgGhuxycl37mbLum/j9C0jJP+k/swHBB+qEblNyMcdKVyDdJwz6GckSvstIlzjnP3KJ/keTshzT5nMoHYE+NhQ84tANwxYkbZ3eFjqrG7EGD+QDouAcmZlaG6fdM6XbKLRU7XYZI4sjn3LAjt9K5xTKitxbjIj7ZnmwGO26k6cRdZUSM+SJJ4Pmmyiy4wFewYInWLe21+gs+b/3v4drS9XzSAVoirkeEJ8piD324sABJbhfP5+37HRATAb6vPKGl8NG7IzhFCuv1Ygd9GzYlPZ/eT//KkYL5a88E8o4cbCrGZRU70EsZGjBENSNeNC0yq49BW5w1Ml7aIWAC6H73UPe5ZLMTYfZVS0d7vZXw8ckzgH+j7U+hPOzDJnv5tQoB8LBVQIvE6D/ixPOWrI9JtxnkxGBRY+DLfYqpvZd5SlihBRIZi0b9gKht8q73gz7sUjGafYD7mlWGmIamUHzDAdTkShla+pDb+uwu02BU45izip5glnbua/6O3BL8voRUHLtGwvWL9K71WYQjlCjvISPgRh3fAcMQKneZQ2law= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6e4e0405-7df8-4d98-055f-08dcbd3c8d29 X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB7763.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Aug 2024 15:11:34.4977 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: I7X6qejelVoqfoseLkmQuK1oXVBfGoK8hA/OonVnJvvblp49TVse+P18MA3cR990 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8146 X-Stat-Signature: hrccg15ozjsn7e68bpmfywytdscp95i6 X-Rspamd-Queue-Id: 106E180021 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1723734702-602866 X-HE-Meta: U2FsdGVkX18OS46vDW11DAR6t2u6x1soegOH4SYjVMpAGqFbDUnj4ZcpnfXGrkSageaLtL3y+UNbkRJsDrUC0XYicVnOw9tiQ6cwkmdRozYXM8sf971INcKNrI44CDDj7fdV2lNldlcd0/8K6+vJE3gC4w+mQvugrtwmwPxo6RWKIsMuOAkzzg5bLB6sxFGCpOkbM629CW6fnKyVyBtO8JQ6x+AO47tt/1/smeus0JU0TOIQqr/OiB0eXWebsKCv2ht1zaOh1ICEx7hngd6zbH5EEH+HeWrGoKRFqnAHqr9s8tZazaC4SE+7s+8rRD3z5l0SzufCfGktzVlonhoyhaRx7TQDf89vrG54PIjkUqT29G/CxQLw4BqGqwRYt9MB9882FNVirBeDIQWbiLCtLlzJ0DV7BwMMk2j/EYJdoE1Jfhf6JTYmEnBcru1wBOLNAJMgNf6ucsLZSwbAF0D0W4wue8aMrnVyxWNpSeQyUhK83f/LiF9+6FX9nbDD61tR3l5wE6STwN7rfFBFwM917rcfBHDDwIafDCDKDrFdorRxZ1wkOInWTrqadh0kyn0RTGikkquHkYKvikMx0vDTpK+Lt2xcgz7zyeHW8z3eb/5TKb6KDkSDHu6mBV7VRyn4JGd0CEGaNGw7Ila7PVQ4A0J1Z/fcjqAsupkOGGCV6AKCozP1uoNmgGn7a7HqUKbYJygzfDeDdeeFUg4yPhjFpoRVPzQ5kxEX703Bj56VgvPNX16uL746PLgGcIEZp/mmuSfoB5ptutW0+BxCiL4SfeoXuOOXtuBS+aBzFmW8641bxv2rYItr1qfnBISvBxiV0kTtIYnZ2kdvg9tUeweKRGIiT9ph91VGiEZpKhT1FyQr/3o4WumpS1tRB7sPuoqhhSkaeaLb4DYT8dABoeDvqkSrpkZvysW9Isq9bXN3WWsmNUotflgEK09vYL4Q/uKWrRK9XkHCZrHU3qCJyOi yUgpzVhL NZaiGBqDrNphb01bck/wGd8YDVllhp/MUqOiWQ7Pu817iFPDdXmv4BurPOZlWy9cik/YdHFtTvmafYpmzUX6beNBqmd8gzxlI33/Shih8mhxZc9ssiZDt4APMigLsV0iM4WwpvyGA4tjUJuZzauJns52b/9tescQD5HwR8NmZvmd2i6uQsPqN7mGCoge3HoReaAZi9Ceg1W2ZE7ZbtB/VIr3VIQFIOm5cL1Ktw1e/If5pD2IoMS9KVgGpED0c+qRw9JD6p7OnmZ1cPZwHyEBuT80NKnUGgs4R1Hvj5JwYJk+kAA6RZGu2GMjSSB0f7XMrws6qG1RFYuFtS6jsV4K5aIVyQtLlIFITfREZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: IOMMU HW now supports updating a dirty bit in an entry when a DMA writes to the entry's VA range. iommufd has a uAPI to read and clear the dirty bits from the tables. This is a trivial recrusive descent algorithm unwound into a function call waterfall. The format needs a function to tell if a contiguous entry is dirty, and a function to clear a contiguous entry back to clean. FIXME: needs kunit testing Signed-off-by: Jason Gunthorpe --- drivers/iommu/generic_pt/iommu_pt.h | 63 +++++++++++++++++++++++++++++ include/linux/generic_pt/iommu.h | 22 ++++++++++ 2 files changed, 85 insertions(+) diff --git a/drivers/iommu/generic_pt/iommu_pt.h b/drivers/iommu/generic_pt/iommu_pt.h index 4fccdcd58d4ba6..79b0ecbdc1adf6 100644 --- a/drivers/iommu/generic_pt/iommu_pt.h +++ b/drivers/iommu/generic_pt/iommu_pt.h @@ -130,6 +130,64 @@ static phys_addr_t NS(iova_to_phys)(struct pt_iommu *iommu_table, return res; } +struct pt_iommu_dirty_args { + struct iommu_dirty_bitmap *dirty; + unsigned int flags; +}; + +/* FIXME this is a bit big on formats with contig.. */ +static __always_inline int +__do_read_and_clear_dirty(struct pt_range *range, void *arg, unsigned int level, + struct pt_table_p *table, pt_level_fn_t descend_fn) +{ + struct pt_state pts = pt_init(range, level, table); + struct pt_iommu_dirty_args *dirty = arg; + + for_each_pt_level_item(&pts) { + if (pts.type == PT_ENTRY_TABLE) + return pt_descend(&pts, arg, descend_fn); + if (pts.type == PT_ENTRY_EMPTY) + continue; + + if (!pt_entry_write_is_dirty(&pts)) + continue; + + /* FIXME we should probably do our own gathering? */ + iommu_dirty_bitmap_record(dirty->dirty, range->va, + log2_to_int(pt_entry_oa_lg2sz(&pts))); + if (!(dirty->flags & IOMMU_DIRTY_NO_CLEAR)) { + /* + * No write log required because DMA incoherence and + * atomic dirty tracking bits can't work together + */ + pt_entry_set_write_clean(&pts); + } + break; + } + return 0; +} +PT_MAKE_LEVELS(__read_and_clear_dirty, __do_read_and_clear_dirty); + +static int __maybe_unused NS(read_and_clear_dirty)( + struct pt_iommu *iommu_table, dma_addr_t iova, dma_addr_t len, + unsigned long flags, struct iommu_dirty_bitmap *dirty_bitmap) +{ + struct pt_iommu_dirty_args dirty = { + .dirty = dirty_bitmap, + .flags = flags, + }; + struct pt_range range; + int ret; + + ret = make_range(common_from_iommu(iommu_table), &range, iova, len); + if (ret) + return ret; + + ret = pt_walk_range(&range, __read_and_clear_dirty, &dirty); + PT_WARN_ON(ret); + return ret; +} + struct pt_iommu_collect_args { struct pt_radix_list_head free_list; u8 ignore_mapped : 1; @@ -887,6 +945,9 @@ static const struct pt_iommu_ops NS(ops) = { .unmap_pages = NS(unmap_pages), .iova_to_phys = NS(iova_to_phys), .cut_mapping = NS(cut_mapping), +#if IS_ENABLED(CONFIG_IOMMUFD_DRIVER) && defined(pt_entry_write_is_dirty) + .read_and_clear_dirty = NS(read_and_clear_dirty), +#endif .get_info = NS(get_info), .deinit = NS(deinit), }; @@ -963,5 +1024,7 @@ EXPORT_SYMBOL_NS_GPL(pt_iommu_init, GENERIC_PT_IOMMU); MODULE_LICENSE("GPL"); MODULE_IMPORT_NS(GENERIC_PT); +/* For iommu_dirty_bitmap_record() */ +MODULE_IMPORT_NS(IOMMUFD); #endif diff --git a/include/linux/generic_pt/iommu.h b/include/linux/generic_pt/iommu.h index d83f293209fa77..f77f6aef3f5958 100644 --- a/include/linux/generic_pt/iommu.h +++ b/include/linux/generic_pt/iommu.h @@ -10,6 +10,7 @@ struct iommu_iotlb_gather; struct pt_iommu_ops; +struct iommu_dirty_bitmap; /** * DOC: IOMMU Radix Page Table @@ -158,6 +159,27 @@ struct pt_iommu_ops { phys_addr_t (*iova_to_phys)(struct pt_iommu *iommu_table, dma_addr_t iova); + /** + * read_and_clear_dirty() - Manipulate the HW set write dirty state + * @iommu_table: Table to manipulate + * @iova: IO virtual address to start + * @size: Length of the IOVA + * @flags: A bitmap of IOMMU_DIRTY_NO_CLEAR + * + * Iterate over all the entries in the mapped range and record their + * write dirty status in iommu_dirty_bitmap. If IOMMU_DIRTY_NO_CLEAR is + * not specified then the entries will be left dirty, otherwise they are + * returned to being not write dirty. + * + * Context: The caller must hold a read range lock that includes @iova. + * + * Returns: -ERRNO on failure, 0 on success. + */ + int (*read_and_clear_dirty)(struct pt_iommu *iommu_table, + dma_addr_t iova, dma_addr_t len, + unsigned long flags, + struct iommu_dirty_bitmap *dirty_bitmap); + /** * get_info() - Return the pt_iommu_info structure * @iommu_table: Table to query