From patchwork Thu Mar 9 10:53:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolin Chen X-Patchwork-Id: 13167205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A3AE5C61DA4 for ; Thu, 9 Mar 2023 10:55:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:Subject:CC :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=tECX5GKy/jW2QPyBADKsVYBNofK2W3r+nZlcgWMoHa4=; b=f2Zp/ZvR5kMgX5 pQqYMW1lSiFGEnWIaTTr94rU8mBKpB14JzYVT1KkiXN4MDqTbOTO8fBd8gCDNMliN327dIe3O/1Uu q0YKgt5tjZn/KgJ2HAzbLahCn8NfjtjcQyLmKWsBmGLRO3OD6JoDdpVOAtuxVUVja2pPhC1s1a/jd Ou0cVQJBWLWaZkYC+xlyiknbFrTf2Dln1KCHfKhI2BT+nn7UYfggDNkaBKZyfS4G4Z8JfGNvP7ddJ /6EmXFJ38rKbMeYZi6f7GGJZK0ltEb4ACOApOReUPrY3k1R3stlkxmLIu0aAGhFlSTcdT4yWVN8Is B99POkTVWBx29BZaOojQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1paDv7-009FBu-3f; Thu, 09 Mar 2023 10:54:49 +0000 Received: from mail-dm6nam11on20629.outbound.protection.outlook.com ([2a01:111:f400:7eaa::629] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1paDv3-009FAL-Iz for linux-arm-kernel@lists.infradead.org; Thu, 09 Mar 2023 10:54:47 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QyEVzyegBGuoCz8RorXkZBJIoSH8baR1J/72YxFWNm2mX2NGedtxLMwkCmswl8wp2xG25wKFGzb0jAgYfuXF6QQl2yZzV7asSys6fGASvDKPXshP7f+FD8Zz7QbCuh4qz/Db58QjZtNwniB0yomJYZb1Jl38Oxjnm2MMTzJQ2N4Hd+BzTesFBiXhpjXC+zF0hsWoEoRrsOt/NObUgpwMeuunrPaj9RS0M3t40b80Iy71zBdAs8cEQugdCmgU+b+6uE9rcNzC0n8wWZHKhzxQMnrGHqON1Cyaicv/M5rLoRT9N530OFBVoELscPZ3n2EeeksNbSAVu9iKSvauhHYF0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Zmoi4KBsEm3qJEPXGjA0t/NaPSxtMhS24o6WqAYI6cU=; b=Frjjc1UK5f4fb8Ej38b2lVd/KmkNrLv+1cvf7IxbWLzVuHoORJnAwafJTgYDGm+Dmo3teoZb0ynkysrJ5aAag57pgJQoj7reunBWAJoQrAzUUGrv3vAy4s1A2JvufYyIVsE90yy+ZtR75GK7dkdbAHCAG7ujKvfAKTBPCkaSXiPsBKlX2WcEbGYRTjqcKxewmyavKUQLPTV8L+LyuMDYazweq9TomUKtOhg/oxnnj/eHILLZpLJ8buK/5YFQB2p/ixMFE1rbxpSvgchZ/6REyRVkhhyhpG8F9iA4lB/WIk321DG0iM/B0cgRo2n920WNLZrgSnTSGQ5lvw0g5TvACQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=arm.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Zmoi4KBsEm3qJEPXGjA0t/NaPSxtMhS24o6WqAYI6cU=; b=sOS5WwxqPSaKlWS9Wu3Lo7mOT1V19Cy5XZPefiTxW3OsmgNeNUX6oEsYiUb6e8wq0n0dWn1ifxDBLP29sWlI62JrGNKldmBd8H0F8cxf5ZYDrFInzPVVm8/hk1EakUSUeZc7MjSGGgr4M2dsuiHM/CXyxZXANyXuqaQLbKiJ6Q45i+LeAwq9XDqFNwItL8OEHso28+6yKeeN9ozooYaVN8F/iCIiMCCsYfQEANbGUdxEs/Y/Q5Fz62eo74iCcmQ0XDkEIqx4aGCFpuoA8TbesqNL9iWSxyH6NECcoIR5+o3neiDgibksDmv6AWRyWUcKp6M+URrNwXiYDOzEdHMk7A== Received: from MN2PR04CA0031.namprd04.prod.outlook.com (2603:10b6:208:d4::44) by CY8PR12MB8297.namprd12.prod.outlook.com (2603:10b6:930:79::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6178.17; Thu, 9 Mar 2023 10:54:40 +0000 Received: from BL02EPF000108E8.namprd05.prod.outlook.com (2603:10b6:208:d4:cafe::65) by MN2PR04CA0031.outlook.office365.com (2603:10b6:208:d4::44) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6178.19 via Frontend Transport; Thu, 9 Mar 2023 10:54:40 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by BL02EPF000108E8.mail.protection.outlook.com (10.167.241.201) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6178.13 via Frontend Transport; Thu, 9 Mar 2023 10:54:39 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Thu, 9 Mar 2023 02:54:33 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Thu, 9 Mar 2023 02:54:32 -0800 Received: from Asurada-Nvidia.nvidia.com (10.127.8.11) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Thu, 9 Mar 2023 02:54:32 -0800 From: Nicolin Chen To: , , CC: , , , , , , , , Subject: [PATCH v1 00/14] Add Nested Translation Support for SMMUv3 Date: Thu, 9 Mar 2023 02:53:36 -0800 Message-ID: X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF000108E8:EE_|CY8PR12MB8297:EE_ X-MS-Office365-Filtering-Correlation-Id: cb89da64-aadb-453f-e781-08db208cae8c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: lo3IEhZlm+i1LLgRF79amTzo3Of0jJMzkO1harKAv1svd+qGb7+BvrU7K4FDg7p0Ztt1+QyexkpLK5XD4pV7q0GeYzDMrohzZ1cgt1NghUcFxHjordusQEO6JXBvw7EqvNsNrMO7qE2GHr2yPQyJUn+zgpLoLxi38L+RCfO9HXrfwVvYb/VUDVwQXNldNzPucGo0X+K3OwBbolk8OS0OIW08rNV5mUaJ/LLr99HrJKjqLtrydKUhdNewp7RDNoTq0mWu3ZD2MwMPraCFi5OVvWGfbY+InUidTajnVbaJLzI83IsW8KBPTKPYhAMk2GyGcdKQ2Aqz70fD3UMlVe7+f1gLB8drvzbj5eLJufNYyTOncku19vt0Q6S43A6NL3bKkLEYoVuroJh0k6ZnMp081j06HleHcPROwnKSKXQG1703p4xY4oa/z2eXTbkRl+JTNrKvaPZEpTDBTZ11DhQ9EeYsIe2j8JVZdqW9LVXRXzuT5K3hL0GZeTCEZgHx/e897CZvJeNMSju7T1XgWZ3BsXJDMIuDNFqVGdzwWfTzblkYJBNww8vhi1FSscikWrJHOb81dQkftSMo9mlne4jrnYl2odAbThAHtgP/AVcRMQqdWF0CfR9eynJ/gA6Xuby/utPvK1WXXNNdp8axIKVbe2B1SF79ivY+YbHgzBZj3pG7YS5blKYkPb7XVq7vEvmV6NWgExDA11lkw+weMKjAeIdw96KIuiiEVWjhZf7RuplIuznrsIanmtCJNZvCBxC78BXv2qGlNXb1PP6s631pBcwI40uXqcUHMdCicaXHlrTjAeb3DxHMEhZ3jznDDAupctZBRj5AwfI+YIOrSYh3fQ== X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230025)(4636009)(39860400002)(376002)(396003)(136003)(346002)(451199018)(40470700004)(46966006)(36840700001)(82740400003)(47076005)(426003)(36860700001)(26005)(36756003)(86362001)(478600001)(966005)(316002)(110136005)(54906003)(356005)(82310400005)(186003)(336012)(41300700001)(83380400001)(40460700003)(7636003)(2616005)(5660300002)(70586007)(40480700001)(7416002)(7696005)(8936002)(8676002)(2906002)(70206006)(4326008);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Mar 2023 10:54:39.7162 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cb89da64-aadb-453f-e781-08db208cae8c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF000108E8.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8297 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230309_025446_020696_96509A8F X-CRM114-Status: GOOD ( 15.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi all, This series of patches add nested translation support for ARM SMMUv3. Eric Auger made a huge effort previously with the VFIO uAPIs, and sent his v16 a year ago. Now, the nested translation should follow the new IOMMUFD uAPIs design. So, most of the key features are ported from the privous VFIO solution, and then rebuilt on top of the IOMMUFD nesting infrastructure. This series is rebased on top of the Intel VT-d nesting changes, so as to reduce merge conflicts at the uapi header updates. The essential parts in the driver to support a nested translation are ->hw_info, ->domain_alloc_user and ->invalidate_cache_user ops. So this series fundamentally adds these three functions in the SMMUv3 driver, along with several preparations and cleanups for them. One unique requirement for SMMUv3 nested translation support is the MSI doorbell address translation, which is a 2-stage translation too. And, to working with the ITS driver, an msi_cookie needs to be setup on the kernel-managed domain, the stage-2 domain of the nesting setup. And the same msi_cookie will be fetched, via iommu_get_domain_for_dev(), in the iommu core to allocate and creates IOVA mappings for the MSI doorbell page(s). However, with the nesting design, the device is attached to a user-managed domain, the stage-1 domain. So both the setup and fetching of the msi_cookie would not work at the level of stage-2 domain. Thus, on both sides, the msi_cookie setup and fetching require a redirection of the domain pointer. It's easy to do so in iommufd core, but needs a new op in the iommu core and driver. You can also find this series on the Github: https://github.com/nicolinc/iommufd/commits/iommufd_nesting The kernel branch is tested with this QEMU branch: https://github.com/nicolinc/qemu/commits/wip/iommufd_rfcv3+nesting+smmuv3 Thanks! Nicolin Chen Eric Auger (2): iommu/arm-smmu-v3: Unset corresponding STE fields when s2_cfg is NULL iommu/arm-smmu-v3: Add STRTAB_STE_0_CFG_NESTED for 2-stage translation Nicolin Chen (12): iommu: Add iommu_get_unmanaged_domain helper iommufd: Add nesting related data structures for ARM SMMUv3 iommufd/device: Setup MSI on kernel-managed domains iommu/arm-smmu-v3: Add arm_smmu_hw_info iommu/arm-smmu-v3: Remove ARM_SMMU_DOMAIN_NESTED iommu/arm-smmu-v3: Prepare for nested domain support iommu/arm-smmu-v3: Implement arm_smmu_get_unmanaged_domain iommu/arm-smmu-v3: Pass in user_cfg to arm_smmu_domain_finalise iommu/arm-smmu-v3: Add arm_smmu_domain_alloc_user iommu/arm-smmu-v3: Support IOMMU_DOMAIN_NESTED type of allocations iommu/arm-smmu-v3: Add CMDQ_OP_TLBI_NH_VAA and CMDQ_OP_TLBI_NH_ALL iommu/arm-smmu-v3: Add arm_smmu_cache_invalidate_user drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 267 ++++++++++++++++---- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 +- drivers/iommu/dma-iommu.c | 5 +- drivers/iommu/iommu-priv.h | 15 ++ drivers/iommu/iommufd/device.c | 5 +- drivers/iommu/iommufd/hw_pagetable.c | 4 + drivers/iommu/iommufd/main.c | 1 + include/linux/iommu.h | 2 + include/uapi/linux/iommufd.h | 64 +++++ 9 files changed, 323 insertions(+), 47 deletions(-)