From patchwork Sat Jan 11 03:32:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolin Chen X-Patchwork-Id: 13935750 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 64769E77188 for ; Sat, 11 Jan 2025 03:35:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=BPeyU3lRqM4dpATbzaAdOE36h5QXLbQQIwpWONiGNKQ=; b=Qkfw48/FEjSL4hc97onGkbPwUk QYNyhNGbcf+mUGTjfTFC4QSJZ9CY+WNjKgMCeIies+3CI+RmQJUVOxfV9a72vUqaAiG/20SQEi3e+ eMzdtCXmmsOZmw8jTJHi7V1CU+GkHHvPxnBJ2XWnEoQnz7Yw3/ggxdAIRG160gDafZHqjkMvuuAhL 7h/kOq7xHCyHvSnPiboUsRRqlWpRAHAeIctrZ1PMMGkUL11eCmKsaY9Ks+SvMoUa1ukuPq/JW8DNN AZyw8UT136CNzhDiwGfMesixyWusoiYlqAl90QGQMEjVRtrdQlDOqhhcq6LdbafH/9juYuIQJ5uHS qookukFQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tWSHb-000000003jR-2nPe; Sat, 11 Jan 2025 03:35:31 +0000 Received: from mail-dm6nam10on20627.outbound.protection.outlook.com ([2a01:111:f403:2413::627] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tWSFA-000000002uJ-0Poy for linux-arm-kernel@lists.infradead.org; Sat, 11 Jan 2025 03:33:01 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=nA8jVy1wASRq8xlqwer3PV8VoGEn7BfNr09o/QRaYTANtSyMMmwt7fN5VPxxLG9EEtDJXZZEFQoa5QOguc3E2/+phdLQlDpsT4sG+G4mGSPRLLvO//T/n20ZODdEPo5zDD0maFRDnUck0vlBJVuAZgo1MwbEeD41/cyxTe3dTjF/wJMp2HIZOpqjFONQVV90HEX53kX6MR5GEI+x0U7IjRD2mPVFq/slUfYjmOAauYRjNWAMH8mcRy2PLMIt1WR5TLk6+Zb7+MEjZpOaCi51vD9Xjl9DgfxWfBCUOQ6nf+/CnN8uAoMG8DO8OlrLZpVTq+grdG1cjHjNl52E1xuIvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BPeyU3lRqM4dpATbzaAdOE36h5QXLbQQIwpWONiGNKQ=; b=w+UAcEJLd3ZZivlPaQRHb7OdMg6uT7P7WKU5luAKStX31mIIdZ7XCG4IZQJ895j6svubILj6n5jkVE1t16Z9/5kVuBm3YaF1JeJ9vOhql/vf8IQ+IMs1fC6h382ziCvI+6i1G+JqhtoSgJaJvJ5gaCpbDex+f2M6EtDlpOcXYw1CmpwU6o35mmTJlpbOqkIctU5b9vMwiAzyX4c7J06V4Y0Zu+/KKKnB5gxQyHZ1wy5qmGepXSzz6oauoNInuMT4wGXKL6RKDzkrIqU+qbvaKtExjPzwfbrGLVrMBad2eXbfYXBRDQXPZ79z+33mO5l9K2nEqw0pu+VpuWHIHmKU3A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BPeyU3lRqM4dpATbzaAdOE36h5QXLbQQIwpWONiGNKQ=; b=tBykNP9j4CgxI4Fswodoi04b2nMyuy9O5eXXCYjV9DIVJE1wdHs9uiZw7zYISWAiXsZRILq+aEUnQTXMIoqHgVnZRldKrNaAPyMpnVX44eyRZbbdnkK/1yYAPXSuFL2RnInnJHnoA5TqbDe2vdi2EYAg8Eroi/ZECuPPBkL8Rt0mFakfas280/FRB9fQizWkPjK8IPep10KUdTwIYG5H7BWHrXLjxZ1JCpPjDoZwm1b8C/lnGy0xWGpvgOw9YN2ORFNPDkm2oiRXxdw53ZorA89oN+w/8uHyhVKslJeESfxfu9coIdCm7kH/ijLFWB1vL41+A1Up5zuh90MGzVR3KA== Received: from DM6PR03CA0098.namprd03.prod.outlook.com (2603:10b6:5:333::31) by DM6PR12MB4220.namprd12.prod.outlook.com (2603:10b6:5:21d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8335.12; Sat, 11 Jan 2025 03:32:50 +0000 Received: from DS1PEPF0001709A.namprd05.prod.outlook.com (2603:10b6:5:333:cafe::dc) by DM6PR03CA0098.outlook.office365.com (2603:10b6:5:333::31) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8335.13 via Frontend Transport; Sat, 11 Jan 2025 03:32:50 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by DS1PEPF0001709A.mail.protection.outlook.com (10.167.18.104) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8335.7 via Frontend Transport; Sat, 11 Jan 2025 03:32:50 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 10 Jan 2025 19:32:49 -0800 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 10 Jan 2025 19:32:49 -0800 Received: from Asurada-Nvidia.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 10 Jan 2025 19:32:46 -0800 From: Nicolin Chen To: , , , , , , CC: , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFCv2 01/13] genirq/msi: Store the IOMMU IOVA directly in msi_desc instead of iommu_cookie Date: Fri, 10 Jan 2025 19:32:17 -0800 Message-ID: <671b2128c193fc9ac9af0f4add96f85a785f513a.1736550979.git.nicolinc@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS1PEPF0001709A:EE_|DM6PR12MB4220:EE_ X-MS-Office365-Filtering-Correlation-Id: 1c8dd264-3da3-4ef5-fbe2-08dd31f0a01d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|36860700013|1800799024|82310400026; X-Microsoft-Antispam-Message-Info: 1vonUtL68tFrCzgxyCTJVwkcnt58I0caVgANFtU+8GgBY7DwMy4Y5KyySCTCBWCslMkawxZysZaJJGH5g5EFhQqkgJm72ls0FXeJ+SJM1qwwstlo3QYI0jmmhNnz2Q3eSZIKAH9oAajCLEaK/ru4LQllxZ+yLmbrVWrjXcHPnxEhZB36L1CO0turttedjnzV3r9KVRq/4KccdViI/v7/AuzqFoUsgum78/diXDFg8nE7FJwI61/viqzPzp4QJZxfo3bBTmoucR3P2zBYEiwC3u3Yo6OT2teTJ+MSYFECyy06XR24k35VHzxUyZrrzLs87eTkQ52JtkH2VOjnQvmRZolmsDkua/hZRTwPz+WPqqWU5WK1vae1nuQo6AgoEAS8M/dPOuAsIiG0qf3xbd9SSOMuqI/wVqyJCK4RmssoXcPGT4IcmkgxXXXBbdX78lpBT2l5fml3PqwZXfBwvQV0iH0QQwGwLSGhDI/oEO3Ksmvz7Wb1l6GQvx3oiULy0w3IFjyxGoUbt0wPlvUU8nDJWbCV6pf2nNHi+FgJ6xHvFyaHIuDroz5/2XzYdBLft0yhRtRkcrZuur28HtgyA1hxpOYO/IBfRd3PDDnE0PLKnY2Dl5IUX23guk6U8HsSjkbqdX2Yy3sjyqdvaTy4jPwbThO3Df1Zc5FhRkbEY3iRbILHRyvLvxyu7qXGu5oD8v2/OA/o0QJxEjBJ25zwvPNvlpwNEo6VtUm+9MHsyjlvJXb54vTYG3NkDkTLG2Ji5OKcR9VLVpokoc85Zyn3Pg1pfZDJZ+ehIW2iRmFPpSXSVayyK6Pxos1r5IvPl6/D9tQ6DZ25isU/6ooUI4Ol0q/k8sHYI5R94Go8UgEMAkm1sqJW8tLYQyBv6F0EM4LBsUgJKpbr3zs243vVxJnCnkAM5zp3zUU4jruM0lXE+7uH6nGY0voLMfODGWOQeqWTiTxj/xuLMrps2gHSVpsxSP7+WRmOy/lfkQAlLwS1i2hXRmgB9vdzJRWprDHBpmUgkNFE4Lv3XT0u1AyChy/bhm350zo7hxHIlEtwfZxXiT4dxt9p7bZjRCAr8KqqmMmRRn4wtJiqk1KVYU1qDxcDQEYsi43u7/RWqjyrg12aSjsNjHM4dMBdHHWk4XXQOThW5Z+ZnaTle8n3rENQdMXFlzPz+mJgpgbS/hVwbIoEUO4NEJp0OBEkMWJrdKXU32YSQQxkTUTlehunp8UnumDhSjlg9SVtfQwe0myp1t4umJwjYKE3kE6c/uwY627RfNFDKnKNe6GrcMNRjefmggEscMB3VnAAwN7QCOhFOuzEmKbZgBrX9sOeWnHlbFFTPDrhy6NLP1sQd+LPQftxrblq3lc4hhn6UvIKu9osrZ8rdrqll7yqo829OmeLiy74SoQ7Bpj3qbAtS999b6EoLqF4FeLnUTpuOyEGbOWbf5pk4ws5F1kLVFMSaRDWtseQD+N8IovhWtOxH/gHbcHGeSqtx6P+fqlF4gJOL6qIjxl4MBiUIhk= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(376014)(36860700013)(1800799024)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jan 2025 03:32:50.3616 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1c8dd264-3da3-4ef5-fbe2-08dd31f0a01d X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS1PEPF0001709A.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4220 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250110_193300_139023_91B79C0B X-CRM114-Status: GOOD ( 21.00 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jason Gunthorpe All the iommu cases simply want to override the MSI page's address with the IOVA that was mapped through the iommu. This doesn't need a cookie pointer, we just need to store the IOVA and its page size in the msi_desc. Instead provide msi_desc_set_iommu_msi_iova() which allows the IOMMU side to specify the IOVA that the MSI page is placed during iommu_dma_prepare(). This is stored in the msi_desc and then iommu_dma_compose_msi_msg() is a simple inline that sets address_hi/lo. The next patch will correct the naming. This is done because we cannot correctly lock access to group->domain in the atomic context that iommu_dma_compose_msi_msg() is called under. Today the locking miss is tolerable because dma_iommu.c operates under an assumption that the domain does not change while a driver is probed. However iommufd now permits the domain to change while the driver is probed and VFIO userspace can create races with IRQ changes calling iommu_dma_prepare/compose_msi_msg() and changing/freeing the iommu_domain. Removing the pointer, and critically, the call to iommu_get_domain_for_dev() during compose resolves this race. Signed-off-by: Jason Gunthorpe Signed-off-by: Nicolin Chen --- include/linux/iommu.h | 6 ------ include/linux/msi.h | 45 +++++++++++++++++++++++---------------- drivers/iommu/dma-iommu.c | 30 +++++--------------------- 3 files changed, 32 insertions(+), 49 deletions(-) diff --git a/include/linux/iommu.h b/include/linux/iommu.h index 318d27841130..3a4215966c1b 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -1513,7 +1513,6 @@ static inline void iommu_debugfs_setup(void) {} int iommu_get_msi_cookie(struct iommu_domain *domain, dma_addr_t base); int iommu_dma_prepare_msi(struct msi_desc *desc, phys_addr_t msi_addr); -void iommu_dma_compose_msi_msg(struct msi_desc *desc, struct msi_msg *msg); #else /* CONFIG_IOMMU_DMA */ @@ -1529,11 +1528,6 @@ static inline int iommu_dma_prepare_msi(struct msi_desc *desc, phys_addr_t msi_a { return 0; } - -static inline void iommu_dma_compose_msi_msg(struct msi_desc *desc, struct msi_msg *msg) -{ -} - #endif /* CONFIG_IOMMU_DMA */ /* diff --git a/include/linux/msi.h b/include/linux/msi.h index b10093c4d00e..d442b4a69d56 100644 --- a/include/linux/msi.h +++ b/include/linux/msi.h @@ -184,7 +184,8 @@ struct msi_desc { struct msi_msg msg; struct irq_affinity_desc *affinity; #ifdef CONFIG_IRQ_MSI_IOMMU - const void *iommu_cookie; + u64 iommu_msi_iova : 58; + u64 iommu_msi_page_shift : 6; #endif #ifdef CONFIG_SYSFS struct device_attribute *sysfs_attrs; @@ -285,28 +286,36 @@ struct msi_desc *msi_next_desc(struct device *dev, unsigned int domid, #define msi_desc_to_dev(desc) ((desc)->dev) -#ifdef CONFIG_IRQ_MSI_IOMMU -static inline const void *msi_desc_get_iommu_cookie(struct msi_desc *desc) -{ - return desc->iommu_cookie; -} - -static inline void msi_desc_set_iommu_cookie(struct msi_desc *desc, - const void *iommu_cookie) +static inline void msi_desc_set_iommu_msi_iova(struct msi_desc *desc, + u64 msi_iova, + unsigned int page_shift) { - desc->iommu_cookie = iommu_cookie; -} -#else -static inline const void *msi_desc_get_iommu_cookie(struct msi_desc *desc) -{ - return NULL; +#ifdef CONFIG_IRQ_MSI_IOMMU + desc->iommu_msi_iova = msi_iova >> page_shift; + desc->iommu_msi_page_shift = page_shift; +#endif } -static inline void msi_desc_set_iommu_cookie(struct msi_desc *desc, - const void *iommu_cookie) +/** + * iommu_dma_compose_msi_msg() - Apply translation to an MSI message + * @desc: MSI descriptor prepared by iommu_dma_prepare_msi() + * @msg: MSI message containing target physical address + */ +static inline void iommu_dma_compose_msi_msg(struct msi_desc *desc, + struct msi_msg *msg) { -} +#ifdef CONFIG_IRQ_MSI_IOMMU + if (desc->iommu_msi_page_shift) { + u64 msi_iova = desc->iommu_msi_iova + << desc->iommu_msi_page_shift; + + msg->address_hi = upper_32_bits(msi_iova); + msg->address_lo = lower_32_bits(msi_iova) | + (msg->address_lo & + ((1 << desc->iommu_msi_page_shift) - 1)); + } #endif +} int msi_domain_insert_msi_desc(struct device *dev, unsigned int domid, struct msi_desc *init_desc); diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 2a9fa0c8cc00..bf91e014d179 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1815,7 +1815,7 @@ int iommu_dma_prepare_msi(struct msi_desc *desc, phys_addr_t msi_addr) static DEFINE_MUTEX(msi_prepare_lock); /* see below */ if (!domain || !domain->iova_cookie) { - desc->iommu_cookie = NULL; + msi_desc_set_iommu_msi_iova(desc, 0, 0); return 0; } @@ -1827,33 +1827,13 @@ int iommu_dma_prepare_msi(struct msi_desc *desc, phys_addr_t msi_addr) mutex_lock(&msi_prepare_lock); msi_page = iommu_dma_get_msi_page(dev, msi_addr, domain); mutex_unlock(&msi_prepare_lock); - - msi_desc_set_iommu_cookie(desc, msi_page); - if (!msi_page) return -ENOMEM; - return 0; -} -/** - * iommu_dma_compose_msi_msg() - Apply translation to an MSI message - * @desc: MSI descriptor prepared by iommu_dma_prepare_msi() - * @msg: MSI message containing target physical address - */ -void iommu_dma_compose_msi_msg(struct msi_desc *desc, struct msi_msg *msg) -{ - struct device *dev = msi_desc_to_dev(desc); - const struct iommu_domain *domain = iommu_get_domain_for_dev(dev); - const struct iommu_dma_msi_page *msi_page; - - msi_page = msi_desc_get_iommu_cookie(desc); - - if (!domain || !domain->iova_cookie || WARN_ON(!msi_page)) - return; - - msg->address_hi = upper_32_bits(msi_page->iova); - msg->address_lo &= cookie_msi_granule(domain->iova_cookie) - 1; - msg->address_lo += lower_32_bits(msi_page->iova); + msi_desc_set_iommu_msi_iova( + desc, msi_page->iova, + ilog2(cookie_msi_granule(domain->iova_cookie))); + return 0; } static int iommu_dma_init(void)