From patchwork Fri Apr 7 18:05:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13205174 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3AB11C6FD1D for ; Fri, 7 Apr 2023 18:01:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231349AbjDGSB4 (ORCPT ); Fri, 7 Apr 2023 14:01:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43140 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231303AbjDGSBz (ORCPT ); Fri, 7 Apr 2023 14:01:55 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 611946184; Fri, 7 Apr 2023 11:01:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680890513; x=1712426513; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=9tjb6xbQRtW38Psd2PFivPrB5ndPCUqQv+VHHV7YSCI=; b=XofAS/6YE6hvkQZKFMUrsArwgQEqLJDAXNaFTayUh0TdvYl4gAODdzqF 4GvD0X0cozCeu1nlPsbPx0hr2p0eOQFmnFjqNtwdKd0BKRYb62EM9ZgDU yssYZpgfG55tCPIVQ0jd7seGt6kiZhdMGhJr85yDO9TBY5hjdHzqUxGTG 3kTcwF8hnMZDLai5GBNOyQniXXndAj+4PClNKZFtD8nt6y+GxR+/VX6su VU5NLrLKPfwpBxl2cdg14O3h0scen0hWXu+rJt/iGa1O7k/Rvq0RKXuOX 61Xvvf37WHhS8NTtjxwcKkJr8aWnCb9qECC40Ju/rSxwk8GyAAmDbR694 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="343046476" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="343046476" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 11:01:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="776910038" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="776910038" Received: from srinivas-otcpl-7600.jf.intel.com (HELO jacob-builder.jf.intel.com) ([10.54.39.106]) by FMSMGA003.fm.intel.com with ESMTP; 07 Apr 2023 11:01:49 -0700 From: Jacob Pan To: LKML , iommu@lists.linux.dev, "Robin Murphy" , Jason Gunthorpe , "Lu Baolu" , Joerg Roedel , dmaengine@vger.kernel.org, vkoul@kernel.org Cc: "Will Deacon" , David Woodhouse , Raj Ashok , "Tian, Kevin" , Yi Liu , "Yu, Fenghua" , Dave Jiang , Tony Luck , "Zanussi, Tom" , Jacob Pan Subject: [PATCH v4 1/7] iommu/vt-d: Use non-privileged mode for all PASIDs Date: Fri, 7 Apr 2023 11:05:48 -0700 Message-Id: <20230407180554.2784285-2-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230407180554.2784285-1-jacob.jun.pan@linux.intel.com> References: <20230407180554.2784285-1-jacob.jun.pan@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org Supervisor Request Enable (SRE) bit in a PASID entry is for permission checking on DMA requests. When SRE = 0, DMA with supervisor privilege will be blocked. However, for in-kernel DMA this is not necessary in that we are targeting kernel memory anyway. There's no need to differentiate user and kernel for in-kernel DMA. Let's use non-privileged (user) permission for all PASIDs used in kernel, it will be consistent with DMA without PASID (RID_PASID) as well. Reviewed-by: Lu Baolu Signed-off-by: Jacob Pan --- drivers/iommu/intel/iommu.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 0768dcae90fd..9f737ef55463 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -2338,8 +2338,6 @@ static int domain_setup_first_level(struct intel_iommu *iommu, if (level != 4 && level != 5) return -EINVAL; - if (pasid != PASID_RID2PASID) - flags |= PASID_FLAG_SUPERVISOR_MODE; if (level == 5) flags |= PASID_FLAG_FL5LP; From patchwork Fri Apr 7 18:05:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13205172 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7397C77B61 for ; Fri, 7 Apr 2023 18:01:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231293AbjDGSBy (ORCPT ); Fri, 7 Apr 2023 14:01:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43090 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231290AbjDGSBx (ORCPT ); Fri, 7 Apr 2023 14:01:53 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B86F25B8B; Fri, 7 Apr 2023 11:01:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680890511; x=1712426511; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=JiKWRtMCJrQDoJeK3U/0yd9kaHBuQqQPeP9njgeoB0o=; b=MA8vZ6KXP4pZonY0eJeeUrclpfekJgkyQ6zlfZB+LZhjHjNxTD/K5pct WKT1xqdMXqntr6k8elRPexUPa40Z0Q8BcfXSWD9Gc+EQL7JXKvq2iwnwA d4o5n3wozxii4InyAtFDXGNMvwJWSHgBr8LoN4GrQMHWqMVgWejx7PNDW aN1Ps0hA9+YtOcve5FKx9k46/pa3Bjs7QLS2IV35BQYoAfYJJAgHOGZa2 izzWbYJPWwFS/YzC4q+siG16MIlt2pHexmPJ7IxrPstlb5jTR3A2qxEyX 3VD2xyUEnievFAna6WnM+ul2LHFF4ly+1ukJFsrKvkRDKvQdun6T5jr+Q Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="343046458" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="343046458" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 11:01:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="776910042" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="776910042" Received: from srinivas-otcpl-7600.jf.intel.com (HELO jacob-builder.jf.intel.com) ([10.54.39.106]) by FMSMGA003.fm.intel.com with ESMTP; 07 Apr 2023 11:01:49 -0700 From: Jacob Pan To: LKML , iommu@lists.linux.dev, "Robin Murphy" , Jason Gunthorpe , "Lu Baolu" , Joerg Roedel , dmaengine@vger.kernel.org, vkoul@kernel.org Cc: "Will Deacon" , David Woodhouse , Raj Ashok , "Tian, Kevin" , Yi Liu , "Yu, Fenghua" , Dave Jiang , Tony Luck , "Zanussi, Tom" , Jacob Pan Subject: [PATCH v4 2/7] iommu/vt-d: Remove PASID supervisor request support Date: Fri, 7 Apr 2023 11:05:49 -0700 Message-Id: <20230407180554.2784285-3-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230407180554.2784285-1-jacob.jun.pan@linux.intel.com> References: <20230407180554.2784285-1-jacob.jun.pan@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org There's no more usage, remove PASID supervisor support. Suggested-by: Lu Baolu Reviewed-by: Lu Baolu Signed-off-by: Jacob Pan --- drivers/iommu/intel/pasid.c | 43 ------------------------------------- drivers/iommu/intel/pasid.h | 7 ------ 2 files changed, 50 deletions(-) diff --git a/drivers/iommu/intel/pasid.c b/drivers/iommu/intel/pasid.c index 633e0a4a01e7..c5d479770e12 100644 --- a/drivers/iommu/intel/pasid.c +++ b/drivers/iommu/intel/pasid.c @@ -335,15 +335,6 @@ static inline void pasid_set_fault_enable(struct pasid_entry *pe) pasid_set_bits(&pe->val[0], 1 << 1, 0); } -/* - * Setup the SRE(Supervisor Request Enable) field (Bit 128) of a - * scalable mode PASID entry. - */ -static inline void pasid_set_sre(struct pasid_entry *pe) -{ - pasid_set_bits(&pe->val[2], 1 << 0, 1); -} - /* * Setup the WPE(Write Protect Enable) field (Bit 132) of a * scalable mode PASID entry. @@ -521,23 +512,6 @@ int intel_pasid_setup_first_level(struct intel_iommu *iommu, return -EINVAL; } - if (flags & PASID_FLAG_SUPERVISOR_MODE) { -#ifdef CONFIG_X86 - unsigned long cr0 = read_cr0(); - - /* CR0.WP is normally set but just to be sure */ - if (unlikely(!(cr0 & X86_CR0_WP))) { - pr_err("No CPU write protect!\n"); - return -EINVAL; - } -#endif - if (!ecap_srs(iommu->ecap)) { - pr_err("No supervisor request support on %s\n", - iommu->name); - return -EINVAL; - } - } - if ((flags & PASID_FLAG_FL5LP) && !cap_fl5lp_support(iommu->cap)) { pr_err("No 5-level paging support for first-level on %s\n", iommu->name); @@ -560,10 +534,6 @@ int intel_pasid_setup_first_level(struct intel_iommu *iommu, /* Setup the first level page table pointer: */ pasid_set_flptr(pte, (u64)__pa(pgd)); - if (flags & PASID_FLAG_SUPERVISOR_MODE) { - pasid_set_sre(pte); - pasid_set_wpe(pte); - } if (flags & PASID_FLAG_FL5LP) pasid_set_flpm(pte, 1); @@ -658,12 +628,6 @@ int intel_pasid_setup_second_level(struct intel_iommu *iommu, pasid_set_fault_enable(pte); pasid_set_page_snoop(pte, !!ecap_smpwc(iommu->ecap)); - /* - * Since it is a second level only translation setup, we should - * set SRE bit as well (addresses are expected to be GPAs). - */ - if (pasid != PASID_RID2PASID && ecap_srs(iommu->ecap)) - pasid_set_sre(pte); pasid_set_present(pte); spin_unlock(&iommu->lock); @@ -700,13 +664,6 @@ int intel_pasid_setup_pass_through(struct intel_iommu *iommu, pasid_set_translation_type(pte, PASID_ENTRY_PGTT_PT); pasid_set_fault_enable(pte); pasid_set_page_snoop(pte, !!ecap_smpwc(iommu->ecap)); - - /* - * We should set SRE bit as well since the addresses are expected - * to be GPAs. - */ - if (ecap_srs(iommu->ecap)) - pasid_set_sre(pte); pasid_set_present(pte); spin_unlock(&iommu->lock); diff --git a/drivers/iommu/intel/pasid.h b/drivers/iommu/intel/pasid.h index 20c54e50f533..d6b7d21244b1 100644 --- a/drivers/iommu/intel/pasid.h +++ b/drivers/iommu/intel/pasid.h @@ -41,13 +41,6 @@ #define FLPT_DEFAULT_DID 1 #define NUM_RESERVED_DID 2 -/* - * The SUPERVISOR_MODE flag indicates a first level translation which - * can be used for access to kernel addresses. It is valid only for - * access to the kernel's static 1:1 mapping of physical memory — not - * to vmalloc or even module mappings. - */ -#define PASID_FLAG_SUPERVISOR_MODE BIT(0) #define PASID_FLAG_NESTED BIT(1) #define PASID_FLAG_PAGE_SNOOP BIT(2) From patchwork Fri Apr 7 18:05:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13205175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4C41C77B70 for ; Fri, 7 Apr 2023 18:01:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231270AbjDGSB4 (ORCPT ); Fri, 7 Apr 2023 14:01:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43146 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231318AbjDGSBz (ORCPT ); Fri, 7 Apr 2023 14:01:55 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 74FCCB451; Fri, 7 Apr 2023 11:01:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680890513; x=1712426513; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=V3GEorKv47qptI/FPYdE9VJz/E7KChATznXusYLtsLA=; b=fQzylXXZ9FbdJD02rnPxy7jacVJR3HuqEp+1DAMj/e6k/2k1BzyBLMEh 6BeICKmDumFaou8ay6PaO2S7mGML4zNgOzO6mWqbXcJNXdQNwdDA+3ot6 02I23vIWyvUETg1LkKfZTnnhg2ICO+vM+PZk+WEpabU/lm8Jvtvqiub8s rwjLScU1G7ilppUkdFNp8k5UJnAloETtZ2FcrRfycCjCP3rpaHlqoIwyL uGqjG45a5Fj/8MKH2W4R1VQfmuDUQi83PZHZYy1uS3sBfGIKhEYmI8stk mfIaLPwuKPwIVEyseZCdsP7M9qNeNIeIOVTytz8gdA7cTuz46fJyRhkt+ w==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="343046480" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="343046480" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 11:01:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="776910053" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="776910053" Received: from srinivas-otcpl-7600.jf.intel.com (HELO jacob-builder.jf.intel.com) ([10.54.39.106]) by FMSMGA003.fm.intel.com with ESMTP; 07 Apr 2023 11:01:50 -0700 From: Jacob Pan To: LKML , iommu@lists.linux.dev, "Robin Murphy" , Jason Gunthorpe , "Lu Baolu" , Joerg Roedel , dmaengine@vger.kernel.org, vkoul@kernel.org Cc: "Will Deacon" , David Woodhouse , Raj Ashok , "Tian, Kevin" , Yi Liu , "Yu, Fenghua" , Dave Jiang , Tony Luck , "Zanussi, Tom" , Jacob Pan Subject: [PATCH v4 3/7] iommu: Support allocation of global PASIDs outside SVA Date: Fri, 7 Apr 2023 11:05:50 -0700 Message-Id: <20230407180554.2784285-4-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230407180554.2784285-1-jacob.jun.pan@linux.intel.com> References: <20230407180554.2784285-1-jacob.jun.pan@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org Devices that use Intel ENQCMD to submit work must use global PASIDs in that the PASID are stored in a per CPU MSR. When such device need to submit work for in-kernel DMA with PASID, it must allocate PASIDs from the same global number space to avoid conflict. This patch moves global PASID allocation APIs from SVA to IOMMU APIs. It is expected that device drivers will use the allocated PASIDs to attach to appropriate IOMMU domains for use. Signed-off-by: Jacob Pan Reviewed-by: Lu Baolu --- v4: move dummy functions outside ifdef CONFIG_IOMMU_SVA (Baolu) --- drivers/iommu/iommu-sva.c | 10 ++++------ drivers/iommu/iommu.c | 33 +++++++++++++++++++++++++++++++++ include/linux/iommu.h | 11 +++++++++++ 3 files changed, 48 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index c434b95dc8eb..222544587582 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -9,15 +9,13 @@ #include "iommu-sva.h" static DEFINE_MUTEX(iommu_sva_lock); -static DEFINE_IDA(iommu_global_pasid_ida); /* Allocate a PASID for the mm within range (inclusive) */ static int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max) { int ret = 0; - if (!pasid_valid(min) || !pasid_valid(max) || - min == 0 || max < min) + if (!pasid_valid(min) || !pasid_valid(max) || max < min) return -EINVAL; mutex_lock(&iommu_sva_lock); @@ -28,8 +26,8 @@ static int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t ma goto out; } - ret = ida_alloc_range(&iommu_global_pasid_ida, min, max, GFP_KERNEL); - if (ret < min) + ret = iommu_alloc_global_pasid(min, max); + if (!pasid_valid(ret)) goto out; mm->pasid = ret; ret = 0; @@ -211,5 +209,5 @@ void mm_pasid_drop(struct mm_struct *mm) if (likely(!pasid_valid(mm->pasid))) return; - ida_free(&iommu_global_pasid_ida, mm->pasid); + iommu_free_global_pasid(mm->pasid); } diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 10db680acaed..2a132ff7e3de 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -38,6 +38,7 @@ static struct kset *iommu_group_kset; static DEFINE_IDA(iommu_group_ida); +static DEFINE_IDA(iommu_global_pasid_ida); static unsigned int iommu_def_domain_type __read_mostly; static bool iommu_dma_strict __read_mostly = IS_ENABLED(CONFIG_IOMMU_DEFAULT_DMA_STRICT); @@ -3450,3 +3451,35 @@ struct iommu_domain *iommu_sva_domain_alloc(struct device *dev, return domain; } + +/** + * @brief + * Allocate a PASID from the global number space. + * + * @param min starting range, inclusive + * @param max ending range, inclusive + * @return The reserved PASID on success or IOMMU_PASID_INVALID on failure. + */ +ioasid_t iommu_alloc_global_pasid(ioasid_t min, ioasid_t max) +{ + int ret; + + if (!pasid_valid(min) || !pasid_valid(max) || max < min) + return IOMMU_PASID_INVALID; + + ret = ida_alloc_range(&iommu_global_pasid_ida, min, max, GFP_KERNEL); + if (ret < 0) + return IOMMU_PASID_INVALID; + + return ret; +} +EXPORT_SYMBOL_GPL(iommu_alloc_global_pasid); + +void iommu_free_global_pasid(ioasid_t pasid) +{ + if (WARN_ON(!pasid_valid(pasid))) + return; + + ida_free(&iommu_global_pasid_ida, pasid); +} +EXPORT_SYMBOL_GPL(iommu_free_global_pasid); diff --git a/include/linux/iommu.h b/include/linux/iommu.h index 54f535ff9868..c9720ddc81d2 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -723,6 +723,8 @@ void iommu_detach_device_pasid(struct iommu_domain *domain, struct iommu_domain * iommu_get_domain_for_dev_pasid(struct device *dev, ioasid_t pasid, unsigned int type); +ioasid_t iommu_alloc_global_pasid(ioasid_t min, ioasid_t max); +void iommu_free_global_pasid(ioasid_t pasid); #else /* CONFIG_IOMMU_API */ struct iommu_ops {}; @@ -1089,6 +1091,13 @@ iommu_get_domain_for_dev_pasid(struct device *dev, ioasid_t pasid, { return NULL; } + +static inline ioasid_t iommu_alloc_global_pasid(ioasid_t min, ioasid_t max) +{ + return IOMMU_PASID_INVALID; +} + +static inline void iommu_free_global_pasid(ioasid_t pasid) {} #endif /* CONFIG_IOMMU_API */ /** @@ -1187,6 +1196,7 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_struct *mm); void iommu_sva_unbind_device(struct iommu_sva *handle); u32 iommu_sva_get_pasid(struct iommu_sva *handle); + #else static inline struct iommu_sva * iommu_sva_bind_device(struct device *dev, struct mm_struct *mm) @@ -1202,6 +1212,7 @@ static inline u32 iommu_sva_get_pasid(struct iommu_sva *handle) { return IOMMU_PASID_INVALID; } + static inline void mm_pasid_init(struct mm_struct *mm) {} static inline void mm_pasid_drop(struct mm_struct *mm) {} #endif /* CONFIG_IOMMU_SVA */ From patchwork Fri Apr 7 18:05:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13205176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EA3CC77B61 for ; Fri, 7 Apr 2023 18:01:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231398AbjDGSB5 (ORCPT ); Fri, 7 Apr 2023 14:01:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43146 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231360AbjDGSB4 (ORCPT ); Fri, 7 Apr 2023 14:01:56 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12D95BB99; Fri, 7 Apr 2023 11:01:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680890515; x=1712426515; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=86aFWjB5alp/mq4OwyCmhytbhDjYJbntMbdQCEJ84y8=; b=maUfJqyqrLEjcV18bsbpulY5ufImGse0zpfKWJ6/ZMU0QcToLjngMpja RumrcuUvhBo8ngSF5zFmxsguXq5eqiCk9jxQ/Md+IPR0wnBpMeR6PYVx+ aPsKXPKcTPJEUdxbqJ8bhYv8qPnnGNrD1EH3aw7vIXsqDGYspaxu+5uqp QD1Ltl9q4JFn7BpFlQXITSb981tPPHP5Xip2VTGI+lERkpp+Hv6NRdKRW Hs+eUcNMMT+U/dcTv30eoiKNyFewk6etBKEGTRl2P4NdNXsWukuSqSya0 U0rYKv8JXchaae5Ux9YWHgH974rmOekhmQvBjpLHXrBhwYX0BN9h1Xymv A==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="343046491" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="343046491" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 11:01:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="776910068" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="776910068" Received: from srinivas-otcpl-7600.jf.intel.com (HELO jacob-builder.jf.intel.com) ([10.54.39.106]) by FMSMGA003.fm.intel.com with ESMTP; 07 Apr 2023 11:01:51 -0700 From: Jacob Pan To: LKML , iommu@lists.linux.dev, "Robin Murphy" , Jason Gunthorpe , "Lu Baolu" , Joerg Roedel , dmaengine@vger.kernel.org, vkoul@kernel.org Cc: "Will Deacon" , David Woodhouse , Raj Ashok , "Tian, Kevin" , Yi Liu , "Yu, Fenghua" , Dave Jiang , Tony Luck , "Zanussi, Tom" , Jacob Pan Subject: [PATCH v4 4/7] iommu/vt-d: Reserve RID_PASID from global PASID space Date: Fri, 7 Apr 2023 11:05:51 -0700 Message-Id: <20230407180554.2784285-5-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230407180554.2784285-1-jacob.jun.pan@linux.intel.com> References: <20230407180554.2784285-1-jacob.jun.pan@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org On VT-d platforms, RID_PASID is used for DMA request without PASID. We should not treat RID_PASID special instead let it be allocated from the global PASID number space. Non-zero value can be used in RID_PASID on Intel VT-d. For ARM, AMD and others that _always_ use 0 as RID_PASID, there is no impact in that SVA PASID allocation base is 1. With this change, devices do both DMA with PASID and SVA will not worry about conflicts when it comes to allocating PASIDs for in-kernel DMA. Signed-off-by: Jacob Pan --- drivers/iommu/intel/iommu.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 9f737ef55463..cbb2670f88ca 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -3956,6 +3956,10 @@ int __init intel_iommu_init(void) intel_iommu_enabled = 1; + /* Reserved RID_PASID from the global namespace for legacy DMA */ + WARN_ON(iommu_alloc_global_pasid(PASID_RID2PASID, PASID_RID2PASID) != + PASID_RID2PASID); + return 0; out_free_dmar: From patchwork Fri Apr 7 18:05:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13205178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E90CC77B61 for ; Fri, 7 Apr 2023 18:02:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231441AbjDGSB7 (ORCPT ); Fri, 7 Apr 2023 14:01:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43212 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231290AbjDGSB5 (ORCPT ); Fri, 7 Apr 2023 14:01:57 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E443BBB7; Fri, 7 Apr 2023 11:01:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680890515; x=1712426515; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XIu8pG3dfBoG5twXS+C7UZckidk9Mw8wBu2etiAguq8=; b=jPmEhIbgpOa64Y1vC2Ble3PfsJLFyJtymsqDFiMitHUZzOyz5hQS7pBF tR7ZvxZCfRGNFgysVXC6HraFxyJCuD68QDBOIn76hVU0WViOo4NWgj4JQ actSCy5joe4zXs8PXqmqOesiDbshgQ858Dri31TTdWTGngkWIxnanNbVb K/sD7n39kaFLs9tW4ld2YjfK+EPxsqDAO4sF8OigS5BYpxlyO2suet00a 0wVNKOOyYxz42rOzOjFGuS2fHmZRITdyUtueFykw/jR06do52iHtXRjW1 8bhZ+wZFtWcqI6xx+d7Ps/E8VG+Xm9/yJpZwM5YFYN/j9nRYb4e5shSRz g==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="343046499" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="343046499" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 11:01:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="776910085" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="776910085" Received: from srinivas-otcpl-7600.jf.intel.com (HELO jacob-builder.jf.intel.com) ([10.54.39.106]) by FMSMGA003.fm.intel.com with ESMTP; 07 Apr 2023 11:01:51 -0700 From: Jacob Pan To: LKML , iommu@lists.linux.dev, "Robin Murphy" , Jason Gunthorpe , "Lu Baolu" , Joerg Roedel , dmaengine@vger.kernel.org, vkoul@kernel.org Cc: "Will Deacon" , David Woodhouse , Raj Ashok , "Tian, Kevin" , Yi Liu , "Yu, Fenghua" , Dave Jiang , Tony Luck , "Zanussi, Tom" , Jacob Pan Subject: [PATCH v4 5/7] iommu/vt-d: Make device pasid attachment explicit Date: Fri, 7 Apr 2023 11:05:52 -0700 Message-Id: <20230407180554.2784285-6-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230407180554.2784285-1-jacob.jun.pan@linux.intel.com> References: <20230407180554.2784285-1-jacob.jun.pan@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org Currently, when a device is attached to its DMA domain, RID_PASID is implicitly attached if VT-d is in scalable mode. To prepare for generic PASID-device domain attachment, this patch parameterizes PASID such that all PASIDs are attached explicitly. It will allow code reuse for DMA API with PASID usage and makes no assumptions of the ordering in which PASIDs and device are attached. The same change applies to IOTLB invalidation and removing PASIDs. Extracted common code based on Baolu's patch: Link:https://lore.kernel.org/linux-iommu/20220614034411.1634238-1-baolu.lu@linux.intel.com/ Signed-off-by: Lu Baolu Signed-off-by: Jacob Pan --- drivers/iommu/intel/iommu.c | 153 ++++++++++++++++++++++++++++-------- drivers/iommu/intel/iommu.h | 8 ++ 2 files changed, 128 insertions(+), 33 deletions(-) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index cbb2670f88ca..52b9d0d3a02c 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -278,6 +278,8 @@ static LIST_HEAD(dmar_satc_units); list_for_each_entry(rmrr, &dmar_rmrr_units, list) static void device_block_translation(struct device *dev); +static void intel_iommu_detach_device_pasid(struct iommu_domain *domain, + struct device *dev, ioasid_t pasid); static void intel_iommu_domain_free(struct iommu_domain *domain); int dmar_disabled = !IS_ENABLED(CONFIG_INTEL_IOMMU_DEFAULT_ON); @@ -1365,6 +1367,7 @@ domain_lookup_dev_info(struct dmar_domain *domain, static void domain_update_iotlb(struct dmar_domain *domain) { + struct device_pasid_info *dev_pasid; struct device_domain_info *info; bool has_iotlb_device = false; unsigned long flags; @@ -1376,6 +1379,14 @@ static void domain_update_iotlb(struct dmar_domain *domain) break; } } + + list_for_each_entry(dev_pasid, &domain->dev_pasids, link_domain) { + info = dev_iommu_priv_get(dev_pasid->dev); + if (info->ats_enabled) { + has_iotlb_device = true; + break; + } + } domain->has_iotlb_device = has_iotlb_device; spin_unlock_irqrestore(&domain->lock, flags); } @@ -1486,6 +1497,7 @@ static void __iommu_flush_dev_iotlb(struct device_domain_info *info, static void iommu_flush_dev_iotlb(struct dmar_domain *domain, u64 addr, unsigned mask) { + struct device_pasid_info *dev_pasid; struct device_domain_info *info; unsigned long flags; @@ -1495,6 +1507,39 @@ static void iommu_flush_dev_iotlb(struct dmar_domain *domain, spin_lock_irqsave(&domain->lock, flags); list_for_each_entry(info, &domain->devices, link) __iommu_flush_dev_iotlb(info, addr, mask); + + list_for_each_entry(dev_pasid, &domain->dev_pasids, link_domain) { + /* device TLB is not aware of the use of RID PASID is for DMA w/o PASID */ + if (dev_pasid->pasid == PASID_RID2PASID) + continue; + + info = dev_iommu_priv_get(dev_pasid->dev); + qi_flush_dev_iotlb_pasid(info->iommu, + PCI_DEVID(info->bus, info->devfn), + info->pfsid, dev_pasid->pasid, + info->ats_qdep, addr, + mask); + } + spin_unlock_irqrestore(&domain->lock, flags); +} + +/* + * The VT-d spec requires to use PASID-based-IOTLB Invalidation to + * invalidate IOTLB and the paging-structure-caches for a first-stage + * page table. + */ +static void domain_flush_pasid_iotlb(struct intel_iommu *iommu, + struct dmar_domain *domain, u64 addr, + unsigned long npages, bool ih) +{ + u16 did = domain_id_iommu(domain, iommu); + struct device_pasid_info *dev_pasid; + unsigned long flags; + + spin_lock_irqsave(&domain->lock, flags); + list_for_each_entry(dev_pasid, &domain->dev_pasids, link_domain) + qi_flush_piotlb(iommu, did, dev_pasid->pasid, addr, npages, ih); + spin_unlock_irqrestore(&domain->lock, flags); } @@ -1514,7 +1559,7 @@ static void iommu_flush_iotlb_psi(struct intel_iommu *iommu, ih = 1 << 6; if (domain->use_first_level) { - qi_flush_piotlb(iommu, did, PASID_RID2PASID, addr, pages, ih); + domain_flush_pasid_iotlb(iommu, domain, addr, pages, ih); } else { unsigned long bitmask = aligned_pages - 1; @@ -1584,7 +1629,7 @@ static void intel_flush_iotlb_all(struct iommu_domain *domain) u16 did = domain_id_iommu(dmar_domain, iommu); if (dmar_domain->use_first_level) - qi_flush_piotlb(iommu, did, PASID_RID2PASID, 0, -1, 0); + domain_flush_pasid_iotlb(iommu, dmar_domain, 0, -1, 0); else iommu->flush.flush_iotlb(iommu, did, 0, 0, DMA_TLB_DSI_FLUSH); @@ -1756,6 +1801,7 @@ static struct dmar_domain *alloc_domain(unsigned int type) domain->use_first_level = true; domain->has_iotlb_device = false; INIT_LIST_HEAD(&domain->devices); + INIT_LIST_HEAD(&domain->dev_pasids); spin_lock_init(&domain->lock); xa_init(&domain->iommu_array); @@ -2429,10 +2475,11 @@ static int __init si_domain_init(int hw) return 0; } -static int dmar_domain_attach_device(struct dmar_domain *domain, - struct device *dev) +static int dmar_domain_attach_device_pasid(struct dmar_domain *domain, + struct device *dev, ioasid_t pasid) { struct device_domain_info *info = dev_iommu_priv_get(dev); + struct device_pasid_info *dev_pasid; struct intel_iommu *iommu; unsigned long flags; u8 bus, devfn; @@ -2442,43 +2489,57 @@ static int dmar_domain_attach_device(struct dmar_domain *domain, if (!iommu) return -ENODEV; + dev_pasid = kzalloc(sizeof(*dev_pasid), GFP_KERNEL); + if (!dev_pasid) + return -ENOMEM; + ret = domain_attach_iommu(domain, iommu); if (ret) - return ret; + goto exit_free; + info->domain = domain; + dev_pasid->pasid = pasid; + dev_pasid->dev = dev; spin_lock_irqsave(&domain->lock, flags); - list_add(&info->link, &domain->devices); + if (!info->dev_attached) + list_add(&info->link, &domain->devices); + + list_add(&dev_pasid->link_domain, &domain->dev_pasids); spin_unlock_irqrestore(&domain->lock, flags); /* PASID table is mandatory for a PCI device in scalable mode. */ if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev)) { /* Setup the PASID entry for requests without PASID: */ if (hw_pass_through && domain_type_is_si(domain)) - ret = intel_pasid_setup_pass_through(iommu, domain, - dev, PASID_RID2PASID); + ret = intel_pasid_setup_pass_through(iommu, domain, dev, pasid); else if (domain->use_first_level) - ret = domain_setup_first_level(iommu, domain, dev, - PASID_RID2PASID); + ret = domain_setup_first_level(iommu, domain, dev, pasid); else - ret = intel_pasid_setup_second_level(iommu, domain, - dev, PASID_RID2PASID); + ret = intel_pasid_setup_second_level(iommu, domain, dev, pasid); if (ret) { - dev_err(dev, "Setup RID2PASID failed\n"); + dev_err(dev, "Setup PASID %d failed\n", pasid); device_block_translation(dev); - return ret; + goto exit_free; } } + /* device context already activated, we are done */ + if (info->dev_attached) + goto exit; ret = domain_context_mapping(domain, dev); if (ret) { dev_err(dev, "Domain context map failed\n"); device_block_translation(dev); - return ret; + goto exit_free; } iommu_enable_pci_caps(info); - + info->dev_attached = 1; +exit: return 0; +exit_free: + kfree(dev_pasid); + return ret; } static bool device_has_rmrr(struct device *dev) @@ -4029,8 +4090,7 @@ static void device_block_translation(struct device *dev) iommu_disable_pci_caps(info); if (!dev_is_real_dma_subdevice(dev)) { if (sm_supported(iommu)) - intel_pasid_tear_down_entry(iommu, dev, - PASID_RID2PASID, false); + intel_iommu_detach_device_pasid(&info->domain->domain, dev, PASID_RID2PASID); else domain_context_clear(info); } @@ -4040,6 +4100,7 @@ static void device_block_translation(struct device *dev) spin_lock_irqsave(&info->domain->lock, flags); list_del(&info->link); + info->dev_attached = 0; spin_unlock_irqrestore(&info->domain->lock, flags); domain_detach_iommu(info->domain, iommu); @@ -4186,7 +4247,7 @@ static int intel_iommu_attach_device(struct iommu_domain *domain, if (ret) return ret; - return dmar_domain_attach_device(to_dmar_domain(domain), dev); + return dmar_domain_attach_device_pasid(to_dmar_domain(domain), dev, PASID_RID2PASID); } static int intel_iommu_map(struct iommu_domain *domain, @@ -4675,26 +4736,52 @@ static void intel_iommu_iotlb_sync_map(struct iommu_domain *domain, __mapping_notify_one(info->iommu, dmar_domain, pfn, pages); } -static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid) +static void intel_iommu_detach_device_pasid(struct iommu_domain *domain, + struct device *dev, ioasid_t pasid) { - struct intel_iommu *iommu = device_to_iommu(dev, NULL, NULL); - struct iommu_domain *domain; + struct device_domain_info *info = dev_iommu_priv_get(dev); + struct dmar_domain *dmar_domain = to_dmar_domain(domain); + struct device_pasid_info *i, *dev_pasid = NULL; + struct intel_iommu *iommu = info->iommu; + unsigned long flags; - /* Domain type specific cleanup: */ - domain = iommu_get_domain_for_dev_pasid(dev, pasid, 0); - if (domain) { - switch (domain->type) { - case IOMMU_DOMAIN_SVA: - intel_svm_remove_dev_pasid(dev, pasid); - break; - default: - /* should never reach here */ - WARN_ON(1); + spin_lock_irqsave(&dmar_domain->lock, flags); + list_for_each_entry(i, &dmar_domain->dev_pasids, link_domain) { + if (i->dev == dev && i->pasid == pasid) { + list_del(&i->link_domain); + dev_pasid = i; break; } } + spin_unlock_irqrestore(&dmar_domain->lock, flags); + if (WARN_ON(!dev_pasid)) + return; + + /* PASID entry already cleared during SVA unbind */ + if (domain->type != IOMMU_DOMAIN_SVA) + intel_pasid_tear_down_entry(iommu, dev, pasid, false); + + kfree(dev_pasid); +} + +static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid) +{ + struct device_domain_info *info = dev_iommu_priv_get(dev); + struct dmar_domain *dmar_domain; + struct iommu_domain *domain; + + domain = iommu_get_domain_for_dev_pasid(dev, pasid, 0); + dmar_domain = to_dmar_domain(domain); + + /* + * SVA Domain type specific cleanup: Not ideal but not until we have + * IOPF capable domain specific ops, we need this special case. + */ + if (domain->type == IOMMU_DOMAIN_SVA) + return intel_svm_remove_dev_pasid(dev, pasid); - intel_pasid_tear_down_entry(iommu, dev, pasid, false); + intel_iommu_detach_device_pasid(domain, dev, pasid); + domain_detach_iommu(dmar_domain, info->iommu); } const struct iommu_ops intel_iommu_ops = { diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index 65b15be72878..b6c26f25d1ba 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -595,6 +595,7 @@ struct dmar_domain { spinlock_t lock; /* Protect device tracking lists */ struct list_head devices; /* all devices' list */ + struct list_head dev_pasids; /* all attached pasids */ struct dma_pte *pgd; /* virtual address */ int gaw; /* max guest address width */ @@ -708,6 +709,7 @@ struct device_domain_info { u8 ats_supported:1; u8 ats_enabled:1; u8 dtlb_extra_inval:1; /* Quirk for devices need extra flush */ + u8 dev_attached:1; /* Device context activated */ u8 ats_qdep; struct device *dev; /* it's NULL for PCIe-to-PCI bridge */ struct intel_iommu *iommu; /* IOMMU used by this device */ @@ -715,6 +717,12 @@ struct device_domain_info { struct pasid_table *pasid_table; /* pasid table */ }; +struct device_pasid_info { + struct list_head link_domain; /* link to domain siblings */ + struct device *dev; /* physical device derived from */ + ioasid_t pasid; /* PASID on physical device */ +}; + static inline void __iommu_flush_cache( struct intel_iommu *iommu, void *addr, int size) { From patchwork Fri Apr 7 18:05:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13205177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D055C77B71 for ; Fri, 7 Apr 2023 18:01:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231432AbjDGSB6 (ORCPT ); Fri, 7 Apr 2023 14:01:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43214 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231390AbjDGSB5 (ORCPT ); Fri, 7 Apr 2023 14:01:57 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6C8BDBB8F; Fri, 7 Apr 2023 11:01:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680890515; x=1712426515; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=uSXXj/CzihsdqWSIfoULHyCBzlYtwHCX/EZVzgaBP38=; b=B9pPDXd5A5LkYrD6YXc5hailB3t5/SymGufTyp1BfV3ewuMvESWHqT9D HyQMdGEdWf6mLO3cAjRatfklP9HnBe39mHLkq5tfAH7dTlGIQ7P1PKdXb SkAlyiPzJqNXs/0/z2VToZJ7tKEkZnrXS/0Lykotv3xODCiX8/ilhFWZP C4XueXv3QMWU2qpKJLPjbfb08C50NI9p/J0lZ2t1z6NueQNnOIlJPlNO1 qMeAZ7blygjG8RcILVVTLSP4St1em5sWx7gkxx1Du0Fwb+RsQKdzUc8Hs zXyafJ/nSec9VjjMteiojA2eLnd+cWYY6IHH/6E9P6gyec8sU8kO/Ro4O A==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="343046510" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="343046510" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 11:01:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="776910092" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="776910092" Received: from srinivas-otcpl-7600.jf.intel.com (HELO jacob-builder.jf.intel.com) ([10.54.39.106]) by FMSMGA003.fm.intel.com with ESMTP; 07 Apr 2023 11:01:52 -0700 From: Jacob Pan To: LKML , iommu@lists.linux.dev, "Robin Murphy" , Jason Gunthorpe , "Lu Baolu" , Joerg Roedel , dmaengine@vger.kernel.org, vkoul@kernel.org Cc: "Will Deacon" , David Woodhouse , Raj Ashok , "Tian, Kevin" , Yi Liu , "Yu, Fenghua" , Dave Jiang , Tony Luck , "Zanussi, Tom" , Jacob Pan Subject: [PATCH v4 6/7] iommu/vt-d: Implement set_dev_pasid domain op Date: Fri, 7 Apr 2023 11:05:53 -0700 Message-Id: <20230407180554.2784285-7-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230407180554.2784285-1-jacob.jun.pan@linux.intel.com> References: <20230407180554.2784285-1-jacob.jun.pan@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org Devices that use ENQCMDS to submit work on buffers mapped by DMA API must attach a PASID to the default domain of the device. In preparation for this use case, this patch implements set_dev_pasid() for the default_domain_ops. If the device context has not been set up prior to this call, this will set up the device context in addition to PASID attachment. Signed-off-by: Jacob Pan --- drivers/iommu/intel/iommu.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 52b9d0d3a02c..1ad9c5a4bd8f 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -4784,6 +4784,26 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid) domain_detach_iommu(dmar_domain, info->iommu); } +static int intel_iommu_attach_device_pasid(struct iommu_domain *domain, + struct device *dev, ioasid_t pasid) +{ + struct device_domain_info *info = dev_iommu_priv_get(dev); + struct dmar_domain *dmar_domain = to_dmar_domain(domain); + struct intel_iommu *iommu = info->iommu; + int ret; + + if (!pasid_supported(iommu)) + return -ENODEV; + + ret = prepare_domain_attach_device(domain, dev); + if (ret) + return ret; + + return dmar_domain_attach_device_pasid(dmar_domain, dev, pasid); +} + + + const struct iommu_ops intel_iommu_ops = { .capable = intel_iommu_capable, .domain_alloc = intel_iommu_domain_alloc, @@ -4803,6 +4823,7 @@ const struct iommu_ops intel_iommu_ops = { #endif .default_domain_ops = &(const struct iommu_domain_ops) { .attach_dev = intel_iommu_attach_device, + .set_dev_pasid = intel_iommu_attach_device_pasid, .map_pages = intel_iommu_map_pages, .unmap_pages = intel_iommu_unmap_pages, .iotlb_sync_map = intel_iommu_iotlb_sync_map, From patchwork Fri Apr 7 18:05:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13205179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4347AC77B6C for ; Fri, 7 Apr 2023 18:02:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231521AbjDGSCG (ORCPT ); Fri, 7 Apr 2023 14:02:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43226 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231424AbjDGSB6 (ORCPT ); Fri, 7 Apr 2023 14:01:58 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D510BDCF; Fri, 7 Apr 2023 11:01:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680890516; x=1712426516; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=YWwzO56YlG7jSpZ2Htd1+a3BTw6//fscFm8c8bDJatI=; b=dY6XsioeOvlGpyf47v1NGT2QPhWO4zmQoOxkcnVKxWNssTIiWpogCXN7 yThh6Jy+prgQx41Eav8UQFEOnYvsEgC9Ak2qltRlBF3h4mns7zdRrXSZJ hET5ehnVSJ02fU9z6g/dmKu1m6574qtZkusJspGpzA4fBhn5pqmYQFQjM cADPuESHXniFOorNXmLJCCdl3CYx+mkaHtonlktT0NspLKrOB3yQKNJkw 4U+9Wo9Xn5P0O3od4Jx/rdOd2Sh3D1mCXlxtUz+ya1mKS+mN/QQczA5Dg VKfNIW5lgrejyBa8EB4fcGfZbGrakKk0vSZbyccP3jB+quztJ/svTy8kO g==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="343046519" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="343046519" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 11:01:55 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="776910101" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="776910101" Received: from srinivas-otcpl-7600.jf.intel.com (HELO jacob-builder.jf.intel.com) ([10.54.39.106]) by FMSMGA003.fm.intel.com with ESMTP; 07 Apr 2023 11:01:52 -0700 From: Jacob Pan To: LKML , iommu@lists.linux.dev, "Robin Murphy" , Jason Gunthorpe , "Lu Baolu" , Joerg Roedel , dmaengine@vger.kernel.org, vkoul@kernel.org Cc: "Will Deacon" , David Woodhouse , Raj Ashok , "Tian, Kevin" , Yi Liu , "Yu, Fenghua" , Dave Jiang , Tony Luck , "Zanussi, Tom" , Jacob Pan Subject: [PATCH v4 7/7] dmaengine/idxd: Re-enable kernel workqueue under DMA API Date: Fri, 7 Apr 2023 11:05:54 -0700 Message-Id: <20230407180554.2784285-8-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230407180554.2784285-1-jacob.jun.pan@linux.intel.com> References: <20230407180554.2784285-1-jacob.jun.pan@linux.intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org Kernel workqueues were disabled due to flawed use of kernel VA and SVA API. Now that we have the support for attaching PASID to the device's default domain and the ability to reserve global PASIDs from SVA APIs, we can re-enable the kernel work queues and use them under DMA API. We also use non-privileged access for in-kernel DMA to be consistent with the IOMMU settings. Consequently, interrupt for user privilege is enabled for work completion IRQs. Link:https://lore.kernel.org/linux-iommu/20210511194726.GP1002214@nvidia.com/ Reviewed-by: Dave Jiang Reviewed-by: Fenghua Yu Signed-off-by: Jacob Pan --- drivers/dma/idxd/device.c | 30 ++++---------------- drivers/dma/idxd/init.c | 60 ++++++++++++++++++++++++++++++++++++--- drivers/dma/idxd/sysfs.c | 7 ----- 3 files changed, 61 insertions(+), 36 deletions(-) diff --git a/drivers/dma/idxd/device.c b/drivers/dma/idxd/device.c index 6fca8fa8d3a8..f6b133d61a04 100644 --- a/drivers/dma/idxd/device.c +++ b/drivers/dma/idxd/device.c @@ -299,21 +299,6 @@ void idxd_wqs_unmap_portal(struct idxd_device *idxd) } } -static void __idxd_wq_set_priv_locked(struct idxd_wq *wq, int priv) -{ - struct idxd_device *idxd = wq->idxd; - union wqcfg wqcfg; - unsigned int offset; - - offset = WQCFG_OFFSET(idxd, wq->id, WQCFG_PRIVL_IDX); - spin_lock(&idxd->dev_lock); - wqcfg.bits[WQCFG_PRIVL_IDX] = ioread32(idxd->reg_base + offset); - wqcfg.priv = priv; - wq->wqcfg->bits[WQCFG_PRIVL_IDX] = wqcfg.bits[WQCFG_PRIVL_IDX]; - iowrite32(wqcfg.bits[WQCFG_PRIVL_IDX], idxd->reg_base + offset); - spin_unlock(&idxd->dev_lock); -} - static void __idxd_wq_set_pasid_locked(struct idxd_wq *wq, int pasid) { struct idxd_device *idxd = wq->idxd; @@ -1324,15 +1309,14 @@ int drv_enable_wq(struct idxd_wq *wq) } /* - * In the event that the WQ is configurable for pasid and priv bits. - * For kernel wq, the driver should setup the pasid, pasid_en, and priv bit. - * However, for non-kernel wq, the driver should only set the pasid_en bit for - * shared wq. A dedicated wq that is not 'kernel' type will configure pasid and + * In the event that the WQ is configurable for pasid, the driver + * should setup the pasid, pasid_en bit. This is true for both kernel + * and user shared workqueues. There is no need to setup priv bit in + * that in-kernel DMA will also do user privileged requests. + * A dedicated wq that is not 'kernel' type will configure pasid and * pasid_en later on so there is no need to setup. */ if (test_bit(IDXD_FLAG_CONFIGURABLE, &idxd->flags)) { - int priv = 0; - if (wq_pasid_enabled(wq)) { if (is_idxd_wq_kernel(wq) || wq_shared(wq)) { u32 pasid = wq_dedicated(wq) ? idxd->pasid : 0; @@ -1340,10 +1324,6 @@ int drv_enable_wq(struct idxd_wq *wq) __idxd_wq_set_pasid_locked(wq, pasid); } } - - if (is_idxd_wq_kernel(wq)) - priv = 1; - __idxd_wq_set_priv_locked(wq, priv); } rc = 0; diff --git a/drivers/dma/idxd/init.c b/drivers/dma/idxd/init.c index e6ee267da0ff..fd4560c91296 100644 --- a/drivers/dma/idxd/init.c +++ b/drivers/dma/idxd/init.c @@ -506,14 +506,65 @@ static struct idxd_device *idxd_alloc(struct pci_dev *pdev, struct idxd_driver_d static int idxd_enable_system_pasid(struct idxd_device *idxd) { - return -EOPNOTSUPP; + struct pci_dev *pdev = idxd->pdev; + struct device *dev = &pdev->dev; + struct iommu_domain *domain; + union gencfg_reg gencfg; + ioasid_t pasid; + int ret; + + /* + * Attach a global PASID to the DMA domain so that we can use ENQCMDS + * to submit work on buffers mapped by DMA API. + */ + domain = iommu_get_domain_for_dev(dev); + if (!domain) + return -EPERM; + + pasid = iommu_alloc_global_pasid(0, dev->iommu->max_pasids); + if (!pasid_valid(pasid)) + return -ENOSPC; + + /* + * DMA domain is owned by the driver, it should support all valid + * types such as DMA-FQ, identity, etc. + */ + ret = iommu_attach_device_pasid(domain, dev, pasid); + if (ret) { + dev_err(dev, "failed to attach device pasid %d, domain type %d", + pasid, domain->type); + iommu_free_global_pasid(pasid); + return ret; + } + + /* Since we set user privilege for kernel DMA, enable completion IRQ */ + gencfg.bits = ioread32(idxd->reg_base + IDXD_GENCFG_OFFSET); + gencfg.user_int_en = 1; + iowrite32(gencfg.bits, idxd->reg_base + IDXD_GENCFG_OFFSET); + idxd->pasid = pasid; + + return ret; } static void idxd_disable_system_pasid(struct idxd_device *idxd) { + struct pci_dev *pdev = idxd->pdev; + struct device *dev = &pdev->dev; + struct iommu_domain *domain; + union gencfg_reg gencfg; + + domain = iommu_get_domain_for_dev(dev); + if (!domain) + return; + + iommu_detach_device_pasid(domain, dev, idxd->pasid); + iommu_free_global_pasid(idxd->pasid); - iommu_sva_unbind_device(idxd->sva); + gencfg.bits = ioread32(idxd->reg_base + IDXD_GENCFG_OFFSET); + gencfg.user_int_en = 0; + iowrite32(gencfg.bits, idxd->reg_base + IDXD_GENCFG_OFFSET); idxd->sva = NULL; + idxd->pasid = IOMMU_PASID_INVALID; } static int idxd_probe(struct idxd_device *idxd) @@ -535,8 +586,9 @@ static int idxd_probe(struct idxd_device *idxd) } else { set_bit(IDXD_FLAG_USER_PASID_ENABLED, &idxd->flags); - if (idxd_enable_system_pasid(idxd)) - dev_warn(dev, "No in-kernel DMA with PASID.\n"); + rc = idxd_enable_system_pasid(idxd); + if (rc) + dev_warn(dev, "No in-kernel DMA with PASID. %d\n", rc); else set_bit(IDXD_FLAG_PASID_ENABLED, &idxd->flags); } diff --git a/drivers/dma/idxd/sysfs.c b/drivers/dma/idxd/sysfs.c index 18cd8151dee0..c5561c00a503 100644 --- a/drivers/dma/idxd/sysfs.c +++ b/drivers/dma/idxd/sysfs.c @@ -944,13 +944,6 @@ static ssize_t wq_name_store(struct device *dev, if (strlen(buf) > WQ_NAME_SIZE || strlen(buf) == 0) return -EINVAL; - /* - * This is temporarily placed here until we have SVM support for - * dmaengine. - */ - if (wq->type == IDXD_WQT_KERNEL && device_pasid_enabled(wq->idxd)) - return -EOPNOTSUPP; - input = kstrndup(buf, count, GFP_KERNEL); if (!input) return -ENOMEM;