From patchwork Thu Aug 15 15:11:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13764912 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07BAEC52D7C for ; Thu, 15 Aug 2024 15:15:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9367B8D0006; Thu, 15 Aug 2024 11:15:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8E5248D0005; Thu, 15 Aug 2024 11:15:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 786128D0006; Thu, 15 Aug 2024 11:15:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 54CCF8D0005 for ; Thu, 15 Aug 2024 11:15:14 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id F057E1C504B for ; Thu, 15 Aug 2024 15:15:13 +0000 (UTC) X-FDA: 82454828106.22.F6AFA95 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2074.outbound.protection.outlook.com [40.107.94.74]) by imf08.hostedemail.com (Postfix) with ESMTP id DF9EC16000F for ; Thu, 15 Aug 2024 15:15:10 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=lfnh7nbv; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf08.hostedemail.com: domain of jgg@nvidia.com designates 40.107.94.74 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723734839; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pqIJ5+fLddzVg4CGWYrqo+HDxROaojy29XidMrKBPsk=; b=PyA8ud9eu1zCLzdQ6MziU/w654A+M37eIux8i9Hs+a+7YX/ys5CGS+3fvWSxPp8fXxMv39 OysNLdJx3rtIbtqj4X6/5Lyx8xLQGYm6szcRGpTyjadRwMGFRFl3vhjWC+/+wq8FvkHCOE VYEkCBuBHjFODe7lRvj8CXi2J1WDusc= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1723734839; a=rsa-sha256; cv=pass; b=j6gjFVTqfsOiLG9nhQtbkPqBCznDDhEgU/hdly/tzN4TIyfZt3VyZYeuKEcgGGjYfuJgMf Q98znKXMTqamm02UXpHw5CjFCPfSH2O7rVwcbgVgRQFSWsve/bywbg7Qrro1VEqYMlTYGy 6LAx2rbCvClTfSDc4+NuXB46PhTWLrw= ARC-Authentication-Results: i=2; imf08.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=lfnh7nbv; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf08.hostedemail.com: domain of jgg@nvidia.com designates 40.107.94.74 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=XK+PUYIpQA0hR7cKxsEWkRAzQOvFCJ2HiiYQ8dUrJ/rbCnqnQzIYPNQNa/ZCI7yUeDdNv0/gZnEeVWt7bSKd04j4gEVSbfldHh7q54WRZWkZLuvfv9xxpKcfp87olcKoNcC7dV+A4hL7BZNlQxtOSdLtbUcmS6Me4Zod70MSN4YXoHAuVXT+xBcXtKKRJ9EvmDarvkT6qS+3gBrHedA8nL7MD2dTGHtkVtqzFhPzqtKdPfuRjJKoIyFXRcV3ecKxPPem2yNy37DTXeu1pd3UB6H6YSwTGvsLKWTOqCBX4W4rlOKNce2n1wwrfZ18NqSMDSQj52xdFoihd78rrMaJvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pqIJ5+fLddzVg4CGWYrqo+HDxROaojy29XidMrKBPsk=; b=eNHbvct3Ext7cTGPAwp01grIl5ZcalWz/2GaRgvV/JlZCNexkI7p6xv9XF+V9/UddlM8a8B3Xs7NxLGpnJBoLvY4U0a/DbGPt/s6nWh2fXV1N4mVpCTiPF80kp4QXyeI2pUditl2FGi9pP4vulovLJP2tfD5qAcQdlIzoeNEEWMft7UuvrFsL0OBKo68l/T1bNEhCN9UrrFUDWYaZk5rIjUjLcj9xXNkD5uPzpgnxnMAcBn6EZj5kxmDxjzUMBgVBzIrKGToP3zZxntfkJoqHCwoX67hBggNcle0d9lcZFFVe41aYZ9hbMTYjfihLQAM///c6H5BuDgTPK0gsBn7lQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pqIJ5+fLddzVg4CGWYrqo+HDxROaojy29XidMrKBPsk=; b=lfnh7nbvZveojA+w98i8Zby6N3Bkpq9ByAnSoQCwDcxfeTRQDxZDsQsGNh3Kp4wGu8jOsrd5/EvyTKQmEeMUPx+Ex+USBf9odjxmliTBcn0rTHmWqvNhpM8pyQZlc/olvORmvn6WUtxUer/C1GRHOSKdb90o2qPWwH6ce9oGobIIVk6ZvbS1BNFxds51bN+dBRW2ocq72hFWnWl6qnQ21XyDZjqdktSftfAlxGjcEK6hQ6UthfloTjNm9g5lLCFsfFGB7OwGAfVYDOWuwPS2XMgnlX41c3/867DFWFUCmshuKbRngEEHv3xU2QKBNUGD/SGGTNTj/0jeTPLuLoEUlg== Received: from CH3PR12MB7763.namprd12.prod.outlook.com (2603:10b6:610:145::10) by SN7PR12MB8146.namprd12.prod.outlook.com (2603:10b6:806:323::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7875.17; Thu, 15 Aug 2024 15:11:35 +0000 Received: from CH3PR12MB7763.namprd12.prod.outlook.com ([fe80::8b63:dd80:c182:4ce8]) by CH3PR12MB7763.namprd12.prod.outlook.com ([fe80::8b63:dd80:c182:4ce8%3]) with mapi id 15.20.7875.016; Thu, 15 Aug 2024 15:11:35 +0000 From: Jason Gunthorpe To: Cc: Alejandro Jimenez , Lu Baolu , David Hildenbrand , Christoph Hellwig , iommu@lists.linux.dev, Joao Martins , Kevin Tian , kvm@vger.kernel.org, linux-mm@kvack.org, Pasha Tatashin , Peter Xu , Ryan Roberts , Sean Christopherson , Tina Zhang Subject: [PATCH 05/16] iommupt: Add unmap_pages op Date: Thu, 15 Aug 2024 12:11:21 -0300 Message-ID: <5-v1-01fa10580981+1d-iommu_pt_jgg@nvidia.com> In-Reply-To: <0-v1-01fa10580981+1d-iommu_pt_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR20CA0049.namprd20.prod.outlook.com (2603:10b6:208:235::18) To CH3PR12MB7763.namprd12.prod.outlook.com (2603:10b6:610:145::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR12MB7763:EE_|SN7PR12MB8146:EE_ X-MS-Office365-Filtering-Correlation-Id: 4d69deff-d8be-4258-2a07-08dcbd3c8d29 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|7416014|376014; X-Microsoft-Antispam-Message-Info: 3R4stLRy0bdo6vwkKvfHENmEVgJ2spyWsshcZCeuniioktuk5D0DF85O/eZP1NgJl8sBBu8zEFUR4wLPMmYHS/3AzWns/RCxTr4kBMiTCENYxAn1hACKoKvKNYPkAc1DpeIjdgq6TXkMYafBYjaVHvDtKCNKfMjxDCkuxkGgHKP1SYU+Ld25pf8MhJXkJxYOmA8nJWPvWaDdZEc8hbrUyd7g4nRbl15voCMaeFAuwd4/BoyLZsQtaFDAA6ipQGnn5kzDGCZ658Ia5C6DVjcStmaBeQYlEc0ItBkdvFIY1pUVKUz/pcnyPyZLYhB51Q0QGp6flF17+8IzYnesozJWmfhaBgixO+99uU3OlmqGMT3gkBOubfw8u0m5wF23GpQz6aTyXLuA11i6tIKDcyTBi8aTF0Mvkg/ZHZjPnNXfy6jtXeL/cFoMzkCWgUADZf7BuXJb3Q4I2D/htFFK5na21fJ99FHf8efe4t6ndE5DQrcPk3Fkm0FNVo+oDQpKKSzbaSydlTyjAhP5RQ1N2e3SKk2/Q7Jh9LyASAUwcbsHYDoyU1nEbVOoJHHvKvEQZbqSBNJ9o5WXhhwzTBbgf8Mo/DSGfho12UxPBx+Qzh1LJWD8LNDm/o7jBKlI/NMoEtYXUmnSTpy9NSTWzdhgrnBywe+VAXhL4mVftCnPNX4M4Jki1C2b9aBxjfv97KJIkbqLrvb0rCdwBIWOM6SVTfFGsQUEaOLHBYfiVuhaPkRyFZS1L2PwRC2mjMp1GHgnOJOoVA6aJsI3h6FRazrT/po7FiW9KthC4ipPFbd186kGMTuJGIDsg9aVRar2RZlHv6XWAyrTakIdaDLyAYdCHDmT7PGG7pjnB/8QlBKDCxeENnM6N18VTF7anNUSa1r3ZZIMB/mWVTyGZbzT4UoB5JpoL75d3iz+Sup/tsFRk3yTKObayJErUZAViBqHF3pTBTxaTd5KPtdQUwIfOK7U8etmohqQl6bIeDEHipkinXdW/m94VFyjYWWeebAzZg3dYm+v8rvZY+DJfCFRp2s15BNiwEjqJZlY2pgZZJjGlixdC4jgkUK3PwK5npBIWo9iC8CZ9QVtJRJzswa5f4V3jAFVbFKSCk7VE71NPexu0BsW+X6peX/lMrI1V9Bv8wJFwn9bzdmg+5n2Ye2aentRo0QdpURznyUzois127CTd57j0idnvOmDTagDC8uq/NnQA0OLktoGeNE27ZgJPBz2gIhBsYDadnFEUDUgHTfCfjXnHFlr/GhFozwLFyqmUg/ImlQiCSXzvPJphp5aY2lI0HX4dN4AhJG4U/rp7YXR1l9WrSgOIeuOsAh3vKuXxV04I+LAL7Gsj91TrrC08QuWQ2Uj1A== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR12MB7763.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(7416014)(376014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: N4nSCERX4YPbhUjo4qkhlLT4pOJ32cgSbObx3rOr+PqQM0Aujd6bftlDX9ne0wsGaW9rXRPglJkM/89gnG2t60MufuXAiOKFZ50J1iPWGMvvCmN1DXlnKNGhkhVXXUiIGEcsKcw1V1yrw0yddbjOoOqZf5qQ/b/M0eU5xRPw4x+6R9IJfzDRHPRpyDyKFWF6orMnbyI0Cmc12qdz7jdf6JEBuTW8ar26sCgxO1Y2FhR1rSSFEWZSG8NZRblhZSnkOV/yvur5RbRrW3TwSU+wDhfs90gTTF4v4svkCf6yyGYHIYPHbFnXXEgJZCR7Rc3rRfTg12V0gWfTdpyN+brMAKdfB1LxbFwUTWSuUvErsAQIOwlrYPNjg+tZ+uOebRnhrQPBazMtfqyE5Gemlf0IOSmDpZF2bJ3xWU5LmHMdn65rBalMEFO9Q5NVAPCUsnlXHv6pyVU/URgzWPjR3Lr3iGwFP/x3jnHHZ2FK9WAhHZgUumS8+Ph86GfVsULi8BEQX0sA85rlX0uVBN4g9K8uPzidtvpnrOtAHwcyZlf/c1zBSv4ECUF15x85WvA6rRbZsJdGie6k8IK75lakRsqVrJ/INwRQpuHo/LUTt7GxvPTEx0OB9b0omiM5TKCgVKFZ0Djd0aTYQ5dr3JTelY2CMefE8Dpx2yDgHLg0QntFpe80NkPvq0+6TBwDB87tJrh1gqSDMXA3nJm25zFz8pplfLuTh84hT4vlpgPneZWaBipDx0xhM4/+ndWvmu2VnDbUjyUxIb5lqgZb7SEEnX2ZI6ZR5Fxpn/KFTMgANTSXBG9xDvn47garl9SrjiXKDUJmUhmNVBLqDB4/ws4jKmG5xsu4Uk3typhmx6TApRxSOmvs3FZVvJLL09PDZ+U2ZgqZLXkaiShMuo9QLFeHFgUfUxHYqnH+3o7UuvOgA+WPspw2ADYAEqHXctM8lft/i3lEI/MjOnLt7iy2A3+9Ow5l0Y13oG3Who1jGaSgSZs5DzhKbLXdCwiM/vny74wNx4ul3nkJm3DQBxfEYY/XhHJqWOZuS8h14SwRRdS1deH+ru+AmBMQLfbvwHEU9bOnO+10DWfQwhJIPN6JQmHo6kkNVGpfC9BCq7XuCsYcfYQsW8G0R2R0d8L8IMy8UadOfSAfp+bsE3XFK5uW1rUyz/OvFbmtvaQsu1+po/OzdkQu4REkWba7GdbkTcpgz7UjD3cqDrDmvIfLbZVBvxUeKPBAg4gHq3oTSF7IEm970lkNrGe32REW8QzFxq4e4Ox6nxuBmwXBa4pKKRJ7XLqCGHQclcAyhHAiZuyGJZnxp3UaOxSXhI4vcJiLhQ18ueXKqDN+pokG0Exrn2xMIgQtE2eMR71sHM/CYQmfjH5bvHmmjucpCiiLQ5mNUndyOAo1dLkPjZIty40efH6EKRXyHv4WiOgUlGcHVDQgYS0qEYYdG29aX6VihqINezsKmH5ZBjR2tnKy5wvi7GCjjvS+QR9vsYRGvCnBBQyATKvvUriZXkFgUOnOfCF97h2DfzojnaIITd3JzBq4s0EtXo2cLmXGveMpYdffruhVrYwF46y/o4o= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4d69deff-d8be-4258-2a07-08dcbd3c8d29 X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB7763.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Aug 2024 15:11:34.5865 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: p1lprTzD8GPAQkVY8Dx8d0cWK65+t5sgsFz6m077qn8HEtIH5DrBmZ3Qj55vwqIB X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8146 X-Rspamd-Queue-Id: DF9EC16000F X-Stat-Signature: mcmotnexqu348ketf1cwdbtme6hyeawy X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1723734910-843929 X-HE-Meta: U2FsdGVkX18KyYFSvm6sSzW/L4f973dNn/ZOLiW88dYv4N5AW5Tp2E0m8PF5Joaoow0C8+RSUq0SwEY33FftvOFSV5Xcf2jA9MsNL2KX938nmaTAzp22fvcsMLFK9P/raOGl+hDo+e2/qq5Sq0m7yRD8r+ladwotsf2tOzWqKDVnDE8dDYh0Ehp0yP3GLswtcHmSBKEdNeA4mPv5Ytwbwm60yQw5zlRIECkd2SSrsftWtM2rxhCTBSaPBr/dLkCg7HJD0X4tPNsp+d90l15JRh1nlO+0QnWDPWTggnIrflPnR3vpee8+ghBqYNg2m+/83qo40+1yukuT4+iZNQ3vu9orYwlq0l2Wi0LfjS9kKpv8UFIrmp12JDpjApeoKE/F6loB2VKSsMb/eCi5McCYBshk9wHfgDtq19s9AKRKrdXwLPy+9Kye6nSRYegnI/LMT1tpyLc4n7aFr6/1CzZJcxUuFEn2L3fms01N9wGKad0eC22GiiEyX1qI345ywCQ9nlw1u45PF/2Rhg3GAk++AHVXaMGZK4Sg1Jga+jlu6krMcNozS96Y3K+2POdpeEZPkFBT0T2/HB2Fnc2HPGk4Uoj5maR2wsWYJB1Kc/xbrbPDOUfiGIqUspkYpVFMUgmE6bpG26C7rNiEfS7IhGh2T8WFaKQ4C7cdN+04xIiZ5W7Oo9KOa6YHCJ4MBfoSDDSYep265Hc/Bgb2pAGV58mrzsUbGzN+8kbMwV2D1f/1tUOYAdUGjkXhgjY1txLyoRVVWoRmCvTZpxHUzSgwwU9eXwFdbkEPs60YJAJOL3bxVKMWBL8IjfhNR7KF3XRyVEwq2p4gYSk80nOh+da1MUb89i1d/w0YAU8TRChy5i7nBKDeokahd88iaYGuO3n2tPOXRAyfcNXiK3WHjeKCA/OAu4sCXJ/tFeoczuyqC8ACi/t0nVoRkl3f7pWBYH6eSXtKVX18NpPzGHvTLGhDeVK lq6qnzY7 INmdi7qVZik3hsb2YsvGKTZlVGnahRCliNwSUi50+LSxusa2gs8r/Xm05QApz+3P4cf31XvLCPCfyDDe+XquFk0i5za3eIrelFSUD69OoXrMg2i3pDl4u8EkaKfM9d/1RGbm+xOy2Y7gzTgjTNhmxqHcsCer88I1CEi7w/xkIgfMFqL8hPJ2Dei2jk0kUe9U0SoC8zZaw3sH3jlDfdQzxwNce95KQo13vvP5rxq1IPAWxJSzxoF95jedjvY35+oxPmr3QuZF8vuzE4wpq5NVCoMkqbpOQg9FLIlKd1kxRVxWQEdT8YwhAcLb/qiE8CTG2eviQ42cUu9bXEpl+igyw+W/5ltwMSYD0lTT+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: unmap_pages removes mappings and any fully contained interior tables from the given range. This follows the strict iommu_domain API definition where it does not split up larger page sizes into smaller. The caller must perform unmap only on ranges created by map or it must have somehow otherwise determined safe cut points (eg iommufd/vfio use iova_to_phys to scan for them) A following patch will provide 'cut' which explicitly does the page size split if the HW can support it. unmap is implemented with a recursive descent of the tree. It has an additional cost of checking that the entire VA range is mapped. If the caller provides a VA range that spans an entire table item then the table can be freed as well. Cache incoherent HW is handled by keep tracking of what table memory ranges need CPU cache invalidation at each level and performing that invalidation once when ascending from that level. Currently, the only user I know of for partial unmap is VFIO type 1 v1.0. Signed-off-by: Jason Gunthorpe --- drivers/iommu/generic_pt/iommu_pt.h | 143 ++++++++++++++++++++++++++++ include/linux/generic_pt/iommu.h | 24 +++++ 2 files changed, 167 insertions(+) diff --git a/drivers/iommu/generic_pt/iommu_pt.h b/drivers/iommu/generic_pt/iommu_pt.h index 835c84ea716093..6d1c59b33d02f3 100644 --- a/drivers/iommu/generic_pt/iommu_pt.h +++ b/drivers/iommu/generic_pt/iommu_pt.h @@ -14,6 +14,63 @@ #include #include +#include +#include + +/* + * Keep track of what table items are being written too during mutation + * operations. When the HW is DMA Incoherent these have to be cache flushed + * before they are visible. The write_log batches flushes together and uses a C + * cleanup to make sure the table memory is flushed before walking concludes + * with that table. + * + * There are two notable cases that need special flushing: + * 1) Installing a table entry requires the new table memory (and all of it's + * children) are flushed. + * 2) Installing a shared table requires that other threads using the shared + * table ensure it is flushed before they attempt to use it. + */ +struct iommu_write_log { + struct pt_range *range; + struct pt_table_p *table; + unsigned int start_idx; + unsigned int last_idx; +}; + +static void record_write(struct iommu_write_log *wlog, + const struct pt_state *pts, + unsigned int index_count_lg2) +{ + if (!(PT_SUPPORTED_FEATURES & BIT(PT_FEAT_DMA_INCOHERENT))) + return; + + if (!wlog->table) { + wlog->table = pts->table; + wlog->start_idx = pts->index; + } + wlog->last_idx = + max(wlog->last_idx, + log2_set_mod(pts->index + log2_to_int(index_count_lg2), 0, + index_count_lg2)); +} + +static void done_writes(struct iommu_write_log *wlog) +{ + struct pt_iommu *iommu_table = iommu_from_common(wlog->range->common); + dma_addr_t dma; + + if (!pt_feature(wlog->range->common, PT_FEAT_DMA_INCOHERENT) || + !wlog->table) + return; + + dma = virt_to_phys(wlog->table); + dma_sync_single_for_device(iommu_table->iommu_device, + dma + wlog->start_idx * PT_ENTRY_WORD_SIZE, + (wlog->last_idx - wlog->start_idx + 1) * + PT_ENTRY_WORD_SIZE, + DMA_TO_DEVICE); + wlog->table = NULL; +} static int make_range(struct pt_common *common, struct pt_range *range, dma_addr_t iova, dma_addr_t len) @@ -102,6 +159,91 @@ static int __collect_tables(struct pt_range *range, void *arg, return 0; } +struct pt_unmap_args { + struct pt_radix_list_head free_list; + pt_vaddr_t unmapped; +}; + +static int __unmap_pages(struct pt_range *range, void *arg, unsigned int level, + struct pt_table_p *table) +{ + struct iommu_write_log wlog __cleanup(done_writes) = { .range = range }; + struct pt_state pts = pt_init(range, level, table); + struct pt_unmap_args *unmap = arg; + int ret; + + for_each_pt_level_item(&pts) { + switch (pts.type) { + case PT_ENTRY_TABLE: { + /* descend will change va */ + bool fully_covered = pt_entry_fully_covered( + &pts, pt_table_item_lg2sz(&pts)); + + ret = pt_descend(&pts, arg, __unmap_pages); + if (ret) + return ret; + + /* + * If the unmapping range fully covers the table then we + * can free it as well. The clear is delayed until we + * succeed in clearing the lower table levels. + */ + if (fully_covered) { + pt_radix_add_list(&unmap->free_list, + pts.table_lower); + record_write(&wlog, &pts, ilog2(1)); + pt_clear_entry(&pts, ilog2(1)); + } + break; + } + case PT_ENTRY_EMPTY: + return -EFAULT; + case PT_ENTRY_OA: + /* + * The IOMMU API does not require drivers to support + * unmapping parts of pages. Only legacy VFIO type 1 v1 + * will attempt it after probing for "fine-grained + * superpages" support. There it allows the v1 version + * of VFIO (that nobody uses) to pass more than + * PAGE_SIZE to map. + */ + if (!pt_entry_fully_covered(&pts, + pt_entry_oa_lg2sz(&pts))) + return -EADDRINUSE; + unmap->unmapped += log2_to_int(pt_entry_oa_lg2sz(&pts)); + record_write(&wlog, &pts, + pt_entry_num_contig_lg2(&pts)); + pt_clear_entry(&pts, pt_entry_num_contig_lg2(&pts)); + break; + } + } + return 0; +} + +static size_t NS(unmap_pages)(struct pt_iommu *iommu_table, dma_addr_t iova, + dma_addr_t len, + struct iommu_iotlb_gather *iotlb_gather) +{ + struct pt_common *common = common_from_iommu(iommu_table); + struct pt_unmap_args unmap = {}; + struct pt_range range; + int ret; + + ret = make_range(common_from_iommu(iommu_table), &range, iova, len); + if (ret) + return ret; + + pt_walk_range(&range, __unmap_pages, &unmap); + + if (pt_feature(common, PT_FEAT_DMA_INCOHERENT)) + pt_radix_stop_incoherent_list(&unmap.free_list, + iommu_table->iommu_device); + + /* FIXME into gather */ + pt_radix_free_list_rcu(&unmap.free_list); + return unmap.unmapped; +} + static void NS(get_info)(struct pt_iommu *iommu_table, struct pt_iommu_info *info) { @@ -143,6 +285,7 @@ static void NS(deinit)(struct pt_iommu *iommu_table) } static const struct pt_iommu_ops NS(ops) = { + .unmap_pages = NS(unmap_pages), .iova_to_phys = NS(iova_to_phys), .get_info = NS(get_info), .deinit = NS(deinit), diff --git a/include/linux/generic_pt/iommu.h b/include/linux/generic_pt/iommu.h index 5cd56eac14b41d..bdb6bf2c2ebe85 100644 --- a/include/linux/generic_pt/iommu.h +++ b/include/linux/generic_pt/iommu.h @@ -8,6 +8,7 @@ #include #include +struct iommu_iotlb_gather; struct pt_iommu_ops; /** @@ -60,6 +61,29 @@ struct pt_iommu_info { /* See the function comments in iommu_pt.c for kdocs */ struct pt_iommu_ops { + /** + * unmap_pages() - Make a range of IOVA empty/not present + * @iommu_table: Table to manipulate + * @iova: IO virtual address to start + * @len: Length of the range starting from @iova + * @gather: Gather struct that must be flushed on return + * + * unmap_pages() will remove translation created by map_pages(). + * It cannot subdivide a mapping created by map_pages(), + * so it should be called with IOVA ranges that match those passed + * to map_pages. The IOVA range can aggregate contiguous map_pages() calls + * so long as no individual range is split. + * + * Context: The caller must hold a write range lock that includes + * the whole range. + * + * Returns: Number of bytes of VA unmapped. iova + res will be the + * point unmapping stopped. + */ + size_t (*unmap_pages)(struct pt_iommu *iommu_table, dma_addr_t iova, + dma_addr_t len, + struct iommu_iotlb_gather *iotlb_gather); + /** * iova_to_phys() - Return the output address for the given IOVA * @iommu_table: Table to query