From patchwork Thu Jun 8 09:52:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Duan, Zhenzhong" X-Patchwork-Id: 13271865 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 601BEC7EE2E for ; Thu, 8 Jun 2023 10:06:05 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q7CWP-000059-Or; Thu, 08 Jun 2023 06:05:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q7CWM-0008VC-Az for qemu-devel@nongnu.org; Thu, 08 Jun 2023 06:05:35 -0400 Received: from mga17.intel.com ([192.55.52.151]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q7CWI-0007r9-O3 for qemu-devel@nongnu.org; Thu, 08 Jun 2023 06:05:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1686218730; x=1717754730; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=OaIkrpT4BJ2w/bnS1O5gHV/Ymm37eAt1QkoERXNUudw=; b=TyH2348QyE4Tmn1j+UvjHHH3GwXml4M3a8iG7amoaqWuIMpCABr92Hh2 I+k/CWT8Ndz2o3Ch8eONPHu4g/r5jYh/m6sudeUmI87zn+4Au0uQHTIzW 66Mj5Ycq2y/oQpFxFVPhQH4CmT3xONKxyB1u8sqqtiHfF8HERbMwgwVRe 5JXRLCbhCrm24rhjCblFQrRVu2vCUFQjIO6poV2dSi9sc2OS1l2qPUVqD PHGvIukdx+uxGjaPIcZvEcOpB4y3ccMCM+naE44IYX6yHTf5hrP4A5RRs CcRf0mqIJgR3qA63xA35utwZqzanmertl7oTvtZt5bHjAuBqFvLykhjdh A==; X-IronPort-AV: E=McAfee;i="6600,9927,10734"; a="337624405" X-IronPort-AV: E=Sophos;i="6.00,226,1681196400"; d="scan'208";a="337624405" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2023 03:05:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10734"; a="956652033" X-IronPort-AV: E=Sophos;i="6.00,226,1681196400"; d="scan'208";a="956652033" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2023 03:05:22 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: mst@redhat.com, peterx@redhat.com, jasowang@redhat.com, pbonzini@redhat.com, richard.henderson@linaro.org, eduardo@habkost.net, marcel.apfelbaum@gmail.com, alex.williamson@redhat.com, clg@redhat.com, david@redhat.com, philmd@linaro.org, kwankhede@nvidia.com, cjia@nvidia.com, yi.l.liu@intel.com, chao.p.peng@intel.com Subject: [PATCH v3 1/5] util: Add iova_tree_foreach_range_data Date: Thu, 8 Jun 2023 17:52:27 +0800 Message-Id: <20230608095231.225450-2-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230608095231.225450-1-zhenzhong.duan@intel.com> References: <20230608095231.225450-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=zhenzhong.duan@intel.com; helo=mga17.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This function is a variant of iova_tree_foreach and support tranversing a range to trigger callback with a private data. Reviewed-by: Peter Xu Signed-off-by: Zhenzhong Duan --- include/qemu/iova-tree.h | 17 +++++++++++++++-- util/iova-tree.c | 31 +++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+), 2 deletions(-) diff --git a/include/qemu/iova-tree.h b/include/qemu/iova-tree.h index 8528e5c98fbc..df3dba79e671 100644 --- a/include/qemu/iova-tree.h +++ b/include/qemu/iova-tree.h @@ -39,6 +39,7 @@ typedef struct DMAMap { IOMMUAccessFlags perm; } QEMU_PACKED DMAMap; typedef gboolean (*iova_tree_iterator)(DMAMap *map); +typedef gboolean (*iova_tree_iterator_2)(DMAMap *map, gpointer *private); /** * iova_tree_new: @@ -131,11 +132,23 @@ const DMAMap *iova_tree_find_address(const IOVATree *tree, hwaddr iova); * @iterator: the interator for the mappings, return true to stop * * Iterate over the iova tree. - * - * Return: 1 if found any overlap, 0 if not, <0 if error. */ void iova_tree_foreach(IOVATree *tree, iova_tree_iterator iterator); +/** + * iova_tree_foreach_range_data: + * + * @tree: the iova tree to iterate on + * @range: the iova range to iterate in + * @func: the interator for the mappings, return true to stop + * @private: parameter passed to @func + * + * Iterate over an iova range in iova tree. + */ +void iova_tree_foreach_range_data(IOVATree *tree, DMAMap *range, + iova_tree_iterator_2 func, + gpointer *private); + /** * iova_tree_alloc_map: * diff --git a/util/iova-tree.c b/util/iova-tree.c index 536789797e47..a3cbd5198410 100644 --- a/util/iova-tree.c +++ b/util/iova-tree.c @@ -42,6 +42,12 @@ typedef struct IOVATreeFindIOVAArgs { const DMAMap *result; } IOVATreeFindIOVAArgs; +typedef struct IOVATreeIterator { + DMAMap *range; + iova_tree_iterator_2 func; + gpointer *private; +} IOVATreeIterator; + /** * Iterate args to the next hole * @@ -164,6 +170,31 @@ void iova_tree_foreach(IOVATree *tree, iova_tree_iterator iterator) g_tree_foreach(tree->tree, iova_tree_traverse, iterator); } +static gboolean iova_tree_traverse_range(gpointer key, gpointer value, + gpointer data) +{ + DMAMap *map = key; + IOVATreeIterator *iterator = data; + DMAMap *range = iterator->range; + + g_assert(key == value); + + if (iova_tree_compare(map, range, NULL)) { + return false; + } + + return iterator->func(map, iterator->private); +} + +void iova_tree_foreach_range_data(IOVATree *tree, DMAMap *range, + iova_tree_iterator_2 func, + gpointer *private) +{ + IOVATreeIterator iterator = {range, func, private}; + + g_tree_foreach(tree->tree, iova_tree_traverse_range, &iterator); +} + void iova_tree_remove(IOVATree *tree, DMAMap map) { const DMAMap *overlap; From patchwork Thu Jun 8 09:52:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Duan, Zhenzhong" X-Patchwork-Id: 13271866 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9D54AC7EE23 for ; Thu, 8 Jun 2023 10:06:05 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q7CWa-00007B-Df; Thu, 08 Jun 2023 06:05:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q7CWP-00004z-E0 for qemu-devel@nongnu.org; Thu, 08 Jun 2023 06:05:37 -0400 Received: from mga17.intel.com ([192.55.52.151]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q7CWM-0007rz-0F for qemu-devel@nongnu.org; Thu, 08 Jun 2023 06:05:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1686218734; x=1717754734; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=IjNIqIydqN09clBHDQ6C2od4itkNTcrM6ssdImRA6h8=; b=bL9vsJUCeH7M2EcSWK8hVs4g4QCu5wCasCnwS4fqf3GcoX/CqzQA+akK UubziOuUJhvo6jtxFo0LO34ncT+Ax8RI0Ic2fQHQvb2w8nNgp+D9LEqn1 i1YiZ5aBd1OOWsSEarWOv5agjhXPUaHrhiIvXqhe/975lniFE53w1otPp HkYdHF9PoClwamD6iTy64h6wNTywPXZzMvI3XB2TtwqYvZE+HF7C7TLgp cXZ2zTrjaBudKtJjvzOxAyVKPQQ1OYfz3EMVrtOximCFriKTUT3Gb/JVc euaSFqyxZMz7uaXCOSy3Q6ZsgtSmC87ihgbD+aAJ9l1E/KN8zjfXnjC2k Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10734"; a="337624420" X-IronPort-AV: E=Sophos;i="6.00,226,1681196400"; d="scan'208";a="337624420" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2023 03:05:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10734"; a="956652054" X-IronPort-AV: E=Sophos;i="6.00,226,1681196400"; d="scan'208";a="956652054" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2023 03:05:26 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: mst@redhat.com, peterx@redhat.com, jasowang@redhat.com, pbonzini@redhat.com, richard.henderson@linaro.org, eduardo@habkost.net, marcel.apfelbaum@gmail.com, alex.williamson@redhat.com, clg@redhat.com, david@redhat.com, philmd@linaro.org, kwankhede@nvidia.com, cjia@nvidia.com, yi.l.liu@intel.com, chao.p.peng@intel.com Subject: [PATCH v3 2/5] intel_iommu: Fix a potential issue in VFIO dirty page sync Date: Thu, 8 Jun 2023 17:52:28 +0800 Message-Id: <20230608095231.225450-3-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230608095231.225450-1-zhenzhong.duan@intel.com> References: <20230608095231.225450-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=zhenzhong.duan@intel.com; helo=mga17.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Peter Xu found a potential issue: "The other thing is when I am looking at the new code I found that we actually extended the replay() to be used also in dirty tracking of vfio, in vfio_sync_dirty_bitmap(). For that maybe it's already broken if unmap_all() because afaiu log_sync() can be called in migration thread anytime during DMA so I think it means the device is prone to DMA with the IOMMU pgtable quickly erased and rebuilt here, which means the DMA could fail unexpectedly. Copy Alex, Kirti and Neo." Fix it by replacing the unmap_all() to only evacuate the iova tree (keeping all host mappings untouched, IOW, don't notify UNMAP), and do a full resync in page walk which will notify all existing mappings as MAP. This way we don't interrupt with any existing mapping if there is (e.g. for the dirty sync case), meanwhile we keep sync too to latest (for moving a vfio device into an existing iommu group). Suggested-by: Peter Xu Signed-off-by: Zhenzhong Duan Reviewed-by: Peter Xu --- hw/i386/intel_iommu.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index 94d52f4205d2..34af12f392f5 100644 --- a/hw/i386/intel_iommu.c +++ b/hw/i386/intel_iommu.c @@ -3825,13 +3825,10 @@ static void vtd_iommu_replay(IOMMUMemoryRegion *iommu_mr, IOMMUNotifier *n) IntelIOMMUState *s = vtd_as->iommu_state; uint8_t bus_n = pci_bus_num(vtd_as->bus); VTDContextEntry ce; + DMAMap map = { .iova = 0, .size = HWADDR_MAX }; - /* - * The replay can be triggered by either a invalidation or a newly - * created entry. No matter what, we release existing mappings - * (it means flushing caches for UNMAP-only registers). - */ - vtd_address_space_unmap(vtd_as, n); + /* replay is protected by BQL, page walk will re-setup it safely */ + iova_tree_remove(vtd_as->iova_tree, map); if (vtd_dev_to_context_entry(s, bus_n, vtd_as->devfn, &ce) == 0) { trace_vtd_replay_ce_valid(s->root_scalable ? "scalable mode" : From patchwork Thu Jun 8 09:52:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Duan, Zhenzhong" X-Patchwork-Id: 13271868 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9D94AC7EE29 for ; Thu, 8 Jun 2023 10:06:47 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q7CWi-0000AL-Qw; Thu, 08 Jun 2023 06:05:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q7CWY-00007I-Dk for qemu-devel@nongnu.org; Thu, 08 Jun 2023 06:05:48 -0400 Received: from mga17.intel.com ([192.55.52.151]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q7CWR-0007tA-8C for qemu-devel@nongnu.org; Thu, 08 Jun 2023 06:05:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1686218739; x=1717754739; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=k6pazZM8t2kfEFHaI7eml0v1NUuRM/XJAG7pYSmrhXU=; b=AIV2tIhts6SHvzRjNnpY2g+7X3ZFJ4dj2/daELWG8aA1xLSMLoZOnA3V sqTi8vK9g6M4t5vAvPDNf7+c0Qi7V+BoGsHl/FA3ESzpxoHg/suIHc9F8 vc7y59/oDYL2nnFEQgkehHnT+uWlLQSO7T+GxtGsWkDCsScyBmNJW+Psa 6Fww0cCjVLRr2jKTX3ftVXlLtvkve6ly8d5L75qWtz2A7kczJ6yW28HjZ IIO9K6FmVguLbLqVhRbGgFvFyIkJtM/Y67A07kQuh38x+8ymEDtdR3KyX Mj7bVAYVYOXRGDvhuECaSbnUbubbEM9nl0Yb4rs/efgL65J3uRXrrKkvW Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10734"; a="337624432" X-IronPort-AV: E=Sophos;i="6.00,226,1681196400"; d="scan'208";a="337624432" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2023 03:05:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10734"; a="956652066" X-IronPort-AV: E=Sophos;i="6.00,226,1681196400"; d="scan'208";a="956652066" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2023 03:05:30 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: mst@redhat.com, peterx@redhat.com, jasowang@redhat.com, pbonzini@redhat.com, richard.henderson@linaro.org, eduardo@habkost.net, marcel.apfelbaum@gmail.com, alex.williamson@redhat.com, clg@redhat.com, david@redhat.com, philmd@linaro.org, kwankhede@nvidia.com, cjia@nvidia.com, yi.l.liu@intel.com, chao.p.peng@intel.com Subject: [PATCH v3 3/5] intel_iommu: Fix flag check in replay Date: Thu, 8 Jun 2023 17:52:29 +0800 Message-Id: <20230608095231.225450-4-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230608095231.225450-1-zhenzhong.duan@intel.com> References: <20230608095231.225450-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=zhenzhong.duan@intel.com; helo=mga17.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, T_SCC_BODY_TEXT_LINE=-0.01, T_SPF_TEMPERROR=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Replay doesn't notify registered notifiers but the one passed to it. So it's meaningless to check the registered notifier's synthetic flag. There is no issue currently as all replay use cases have MAP flag set, but let's be robust. Signed-off-by: Zhenzhong Duan Reviewed-by: Peter Xu --- hw/i386/intel_iommu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index 34af12f392f5..f046f8591335 100644 --- a/hw/i386/intel_iommu.c +++ b/hw/i386/intel_iommu.c @@ -3837,7 +3837,7 @@ static void vtd_iommu_replay(IOMMUMemoryRegion *iommu_mr, IOMMUNotifier *n) PCI_FUNC(vtd_as->devfn), vtd_get_domain_id(s, &ce, vtd_as->pasid), ce.hi, ce.lo); - if (vtd_as_has_map_notifier(vtd_as)) { + if (n->notifier_flags & IOMMU_NOTIFIER_MAP) { /* This is required only for MAP typed notifiers */ vtd_page_walk_info info = { .hook_fn = vtd_replay_hook, From patchwork Thu Jun 8 09:52:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Duan, Zhenzhong" X-Patchwork-Id: 13271869 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EE4A4C7EE23 for ; Thu, 8 Jun 2023 10:06:52 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q7CWi-0000A5-9H; Thu, 08 Jun 2023 06:05:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q7CWT-00005a-6R for qemu-devel@nongnu.org; Thu, 08 Jun 2023 06:05:42 -0400 Received: from mga17.intel.com ([192.55.52.151]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q7CWR-0007rz-ON for qemu-devel@nongnu.org; Thu, 08 Jun 2023 06:05:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1686218739; x=1717754739; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=4CgpPYVbeorcoGZpZNNOjXMIcZ32rq+ORgYJFTf1398=; b=EazVifGq4b0QNsmHJ6TV7ol219gQObl9uoTzWEIOJNn1lpAqAd1Tv7rf M8Blf7czpEAUNvPPTcF4CXgWz9SZk3BE189xv9gHpfzIkwfl3r++BLcaD nqQIM1nnnwPQMLvnW3iRLlL0y6OTlVHiPjfXn083xozJNpnXSfYgYU2V7 ogkNMQE2rBq0/uzvgRurlV7BOPokcXMDlu7UDnBM5Diur65qnT/vM4+97 OYd4K1JLP2EgKf558GKNZsP9OrmiIDJiWw2E2YdnpVMBjR7HgmkDzTLzu iSLenj7eGBGh3nF4Mxc9qSR9/prAOA2ukaib3ZlIMSpeomDE1Rp6jofXj Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10734"; a="337624448" X-IronPort-AV: E=Sophos;i="6.00,226,1681196400"; d="scan'208";a="337624448" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2023 03:05:38 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10734"; a="956652087" X-IronPort-AV: E=Sophos;i="6.00,226,1681196400"; d="scan'208";a="956652087" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2023 03:05:34 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: mst@redhat.com, peterx@redhat.com, jasowang@redhat.com, pbonzini@redhat.com, richard.henderson@linaro.org, eduardo@habkost.net, marcel.apfelbaum@gmail.com, alex.williamson@redhat.com, clg@redhat.com, david@redhat.com, philmd@linaro.org, kwankhede@nvidia.com, cjia@nvidia.com, yi.l.liu@intel.com, chao.p.peng@intel.com Subject: [PATCH v3 4/5] intel_iommu: Fix address space unmap Date: Thu, 8 Jun 2023 17:52:30 +0800 Message-Id: <20230608095231.225450-5-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230608095231.225450-1-zhenzhong.duan@intel.com> References: <20230608095231.225450-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=zhenzhong.duan@intel.com; helo=mga17.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org During address space unmap, corresponding IOVA tree entries are also removed. But DMAMap is set beyond notifier's scope by 1, so in theory there is possibility to remove a continuous entry above the notifier's scope but falling in adjacent notifier's scope. There is no issue currently as no use cases allocate notifiers continuously, but let's be robust. Signed-off-by: Zhenzhong Duan Reviewed-by: Peter Xu --- hw/i386/intel_iommu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index f046f8591335..dcc334060cd6 100644 --- a/hw/i386/intel_iommu.c +++ b/hw/i386/intel_iommu.c @@ -3791,7 +3791,7 @@ static void vtd_address_space_unmap(VTDAddressSpace *as, IOMMUNotifier *n) n->start, size); map.iova = n->start; - map.size = size; + map.size = size - 1; /* Inclusive */ iova_tree_remove(as->iova_tree, map); } From patchwork Thu Jun 8 09:52:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Duan, Zhenzhong" X-Patchwork-Id: 13271867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A7287C7EE29 for ; Thu, 8 Jun 2023 10:06:42 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q7CWq-0000B5-Qo; Thu, 08 Jun 2023 06:06:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q7CWp-0000Aw-EY for qemu-devel@nongnu.org; Thu, 08 Jun 2023 06:06:03 -0400 Received: from mga17.intel.com ([192.55.52.151]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q7CWn-0007xy-Kr for qemu-devel@nongnu.org; Thu, 08 Jun 2023 06:06:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1686218761; x=1717754761; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=uB9cWQj7plvWmpBjyLlTfdsBwyWUSZdfHpu6QNZiArs=; b=fVZ1VWvwT2UR2R2zcAQQzFX4K1HSfSYmbmyljbgkEexEC3+057dGC0jU 6zZCXfa5JdCFYpf/8jejUtczD2k7ZZETi0330LYOc2QthOCXasf+uyVQV K7AMzsn1RlzjN3F8wNKaV/3sfyQifY8HXQRjSPE6gAhfJs23kyjjfZwlW TKYihUqKBM4lqPch3qNQy8+Oy6Rfr6YBoa99H04HDAdU1gOH5uFdPbGij XiBT5mrn4SSk9Y4NzN7oAWOjv5mTf3hEySuyfDN3cNryeUsSv4pKnnhkt Xh6lEjkD2Ull8JGlPSCPFSdSfgAMpW9JDwRu0N8hvQr60U7EgKUd7SxtM A==; X-IronPort-AV: E=McAfee;i="6600,9927,10734"; a="337624469" X-IronPort-AV: E=Sophos;i="6.00,226,1681196400"; d="scan'208";a="337624469" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2023 03:05:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10734"; a="956652106" X-IronPort-AV: E=Sophos;i="6.00,226,1681196400"; d="scan'208";a="956652106" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2023 03:05:38 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: mst@redhat.com, peterx@redhat.com, jasowang@redhat.com, pbonzini@redhat.com, richard.henderson@linaro.org, eduardo@habkost.net, marcel.apfelbaum@gmail.com, alex.williamson@redhat.com, clg@redhat.com, david@redhat.com, philmd@linaro.org, kwankhede@nvidia.com, cjia@nvidia.com, yi.l.liu@intel.com, chao.p.peng@intel.com Subject: [PATCH v3 5/5] intel_iommu: Optimize out some unnecessary UNMAP calls Date: Thu, 8 Jun 2023 17:52:31 +0800 Message-Id: <20230608095231.225450-6-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230608095231.225450-1-zhenzhong.duan@intel.com> References: <20230608095231.225450-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.151; envelope-from=zhenzhong.duan@intel.com; helo=mga17.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Commit 63b88968f1 ("intel-iommu: rework the page walk logic") adds logic to record mapped IOVA ranges so we only need to send MAP or UNMAP when necessary. But there is still a corner case of unnecessary UNMAP. During invalidation, either domain or device selective, we only need to unmap when there are recorded mapped IOVA ranges, presuming most of OSes allocating IOVA range continuously, e.g. on x86, linux sets up mapping from 0xffffffff downwards. Strace shows UNMAP ioctl taking 0.000014us and we have 28 such ioctl() in one invalidation, as two notifiers in x86 are split into power of 2 pieces. ioctl(48, VFIO_IOMMU_UNMAP_DMA, 0x7ffffd5c42f0) = 0 <0.000014> The other purpose of this patch is to eliminate noisy error log when we work with IOMMUFD. It looks the duplicate UNMAP call will fail with IOMMUFD while always succeed with legacy container. This behavior difference leads to below error log for IOMMUFD: IOMMU_IOAS_UNMAP failed: No such file or directory vfio_container_dma_unmap(0x562012d6b6d0, 0x0, 0x80000000) = -2 (No such file or directory) IOMMU_IOAS_UNMAP failed: No such file or directory vfio_container_dma_unmap(0x562012d6b6d0, 0x80000000, 0x40000000) = -2 (No such file or directory) ... Suggested-by: Yi Liu Signed-off-by: Zhenzhong Duan --- hw/i386/intel_iommu.c | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index dcc334060cd6..9e5ba81c89e2 100644 --- a/hw/i386/intel_iommu.c +++ b/hw/i386/intel_iommu.c @@ -3743,6 +3743,7 @@ static void vtd_address_space_unmap(VTDAddressSpace *as, IOMMUNotifier *n) hwaddr start = n->start; hwaddr end = n->end; IntelIOMMUState *s = as->iommu_state; + IOMMUTLBEvent event; DMAMap map; /* @@ -3762,22 +3763,25 @@ static void vtd_address_space_unmap(VTDAddressSpace *as, IOMMUNotifier *n) assert(start <= end); size = remain = end - start + 1; + event.type = IOMMU_NOTIFIER_UNMAP; + event.entry.target_as = &address_space_memory; + event.entry.perm = IOMMU_NONE; + /* This field is meaningless for unmap */ + event.entry.translated_addr = 0; + while (remain >= VTD_PAGE_SIZE) { - IOMMUTLBEvent event; uint64_t mask = dma_aligned_pow2_mask(start, end, s->aw_bits); uint64_t size = mask + 1; assert(size); - event.type = IOMMU_NOTIFIER_UNMAP; - event.entry.iova = start; - event.entry.addr_mask = mask; - event.entry.target_as = &address_space_memory; - event.entry.perm = IOMMU_NONE; - /* This field is meaningless for unmap */ - event.entry.translated_addr = 0; - - memory_region_notify_iommu_one(n, &event); + map.iova = start; + map.size = mask; + if (iova_tree_find(as->iova_tree, &map)) { + event.entry.iova = start; + event.entry.addr_mask = mask; + memory_region_notify_iommu_one(n, &event); + } start += size; remain -= size;