From patchwork Thu Jun 15 03:26:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Duan, Zhenzhong" X-Patchwork-Id: 13280674 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D68D5EB64D9 for ; Thu, 15 Jun 2023 03:39:46 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q9dpU-0005cy-1z; Wed, 14 Jun 2023 23:39:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q9dpS-0005cg-G5 for qemu-devel@nongnu.org; Wed, 14 Jun 2023 23:39:22 -0400 Received: from mga03.intel.com ([134.134.136.65]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q9dpQ-0003pb-P2 for qemu-devel@nongnu.org; Wed, 14 Jun 2023 23:39:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1686800360; x=1718336360; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=c+tgk//W5cgwU6zn0AuJ0TG3lRdRmRHblHfLr00w71s=; b=cwmX3f52I5oRPuqjxuAhHZoJTbMgJAGtn4jC5rknvTqGIaUXkMGHhsuT UG6418MBxVP6D2nMulClwpA1EFMFgANV0xdyMLu0iVxphMMcSZSOSikSm sxSzZAyQkmKX74EZp3EyOcMpbaGGcz3GPZzPmp8XKtMoKZhgbuSY0l5a0 gcbcDne7FyD43/TzekFyl6ZG7SBQ35U2NWMMRo4hihsVBA/bcJ9NSkQ9x epfnS59UCCBOSgFIyXuS6+gcBhZXeIlOsy49FZRy8LAH10CBNbgSQDdCc JRvlJ3BsWBItWmuT0Ug/URTkO9oECLG5t4ECT79p/isseOwCVjslYBMv1 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10741"; a="362182799" X-IronPort-AV: E=Sophos;i="6.00,243,1681196400"; d="scan'208";a="362182799" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2023 20:39:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10741"; a="662626510" X-IronPort-AV: E=Sophos;i="6.00,243,1681196400"; d="scan'208";a="662626510" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2023 20:39:15 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: mst@redhat.com, peterx@redhat.com, jasowang@redhat.com, pbonzini@redhat.com, richard.henderson@linaro.org, eduardo@habkost.net, marcel.apfelbaum@gmail.com, alex.williamson@redhat.com, kwankhede@nvidia.com, cjia@nvidia.com, yi.l.liu@intel.com, chao.p.peng@intel.com Subject: [PATCH v4 1/3] intel_iommu: Fix a potential issue in VFIO dirty page sync Date: Thu, 15 Jun 2023 11:26:24 +0800 Message-Id: <20230615032626.314476-2-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230615032626.314476-1-zhenzhong.duan@intel.com> References: <20230615032626.314476-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=134.134.136.65; envelope-from=zhenzhong.duan@intel.com; helo=mga03.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Peter Xu found a potential issue: "The other thing is when I am looking at the new code I found that we actually extended the replay() to be used also in dirty tracking of vfio, in vfio_sync_dirty_bitmap(). For that maybe it's already broken if unmap_all() because afaiu log_sync() can be called in migration thread anytime during DMA so I think it means the device is prone to DMA with the IOMMU pgtable quickly erased and rebuilt here, which means the DMA could fail unexpectedly. Copy Alex, Kirti and Neo." Fix it by replacing the unmap_all() to only evacuate the iova tree (keeping all host mappings untouched, IOW, don't notify UNMAP), and do a full resync in page walk which will notify all existing mappings as MAP. This way we don't interrupt with any existing mapping if there is (e.g. for the dirty sync case), meanwhile we keep sync too to latest (for moving a vfio device into an existing iommu group). Suggested-by: Peter Xu Signed-off-by: Zhenzhong Duan Reviewed-by: Peter Xu --- hw/i386/intel_iommu.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index 94d52f4205d2..34af12f392f5 100644 --- a/hw/i386/intel_iommu.c +++ b/hw/i386/intel_iommu.c @@ -3825,13 +3825,10 @@ static void vtd_iommu_replay(IOMMUMemoryRegion *iommu_mr, IOMMUNotifier *n) IntelIOMMUState *s = vtd_as->iommu_state; uint8_t bus_n = pci_bus_num(vtd_as->bus); VTDContextEntry ce; + DMAMap map = { .iova = 0, .size = HWADDR_MAX }; - /* - * The replay can be triggered by either a invalidation or a newly - * created entry. No matter what, we release existing mappings - * (it means flushing caches for UNMAP-only registers). - */ - vtd_address_space_unmap(vtd_as, n); + /* replay is protected by BQL, page walk will re-setup it safely */ + iova_tree_remove(vtd_as->iova_tree, map); if (vtd_dev_to_context_entry(s, bus_n, vtd_as->devfn, &ce) == 0) { trace_vtd_replay_ce_valid(s->root_scalable ? "scalable mode" : From patchwork Thu Jun 15 03:26:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Duan, Zhenzhong" X-Patchwork-Id: 13280675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 02248EB64D9 for ; Thu, 15 Jun 2023 03:40:07 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q9dpY-0005f8-Mg; Wed, 14 Jun 2023 23:39:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q9dpW-0005eq-OL for qemu-devel@nongnu.org; Wed, 14 Jun 2023 23:39:26 -0400 Received: from mga03.intel.com ([134.134.136.65]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q9dpV-0003qf-69 for qemu-devel@nongnu.org; Wed, 14 Jun 2023 23:39:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1686800365; x=1718336365; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=E3JARyeWHEd/zAisEtngwSFOdTcxaJ6TCBPJvdFo3z0=; b=IriQ5zjCT4ReWIrIHi3oYXZdDbDJMeIRO8kHlX8yALctpXtGBxzKVBqs gfSBP5tra3sLdlgdgVvNqalSXEz928EvqraRnMokLICTkBkWbF8ehxVbQ l6DAXX7GMyoVFJ56CnkCr35reb1d3kIuOu57OfzkOr2RUjzSc9aP8uMhl KUcEJDyypEILJv2fM/C6nGM9AdsiBBk6fY/bBidjnmLhftEq2AWVtHGf3 7sPT3FVRtgEWhjOyrLgqk3bjiE5mNDssfOa4bofBFqij5smWVm/y7aHYe ktVID5auS9HXvcnnAspdMrQKy2oVGuBk+A2akQkLaPkdz2UAH+dYHSaWu g==; X-IronPort-AV: E=McAfee;i="6600,9927,10741"; a="362182811" X-IronPort-AV: E=Sophos;i="6.00,243,1681196400"; d="scan'208";a="362182811" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2023 20:39:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10741"; a="662626519" X-IronPort-AV: E=Sophos;i="6.00,243,1681196400"; d="scan'208";a="662626519" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2023 20:39:20 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: mst@redhat.com, peterx@redhat.com, jasowang@redhat.com, pbonzini@redhat.com, richard.henderson@linaro.org, eduardo@habkost.net, marcel.apfelbaum@gmail.com, alex.williamson@redhat.com, kwankhede@nvidia.com, cjia@nvidia.com, yi.l.liu@intel.com, chao.p.peng@intel.com Subject: [PATCH v4 2/3] intel_iommu: Fix flag check in replay Date: Thu, 15 Jun 2023 11:26:25 +0800 Message-Id: <20230615032626.314476-3-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230615032626.314476-1-zhenzhong.duan@intel.com> References: <20230615032626.314476-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=134.134.136.65; envelope-from=zhenzhong.duan@intel.com; helo=mga03.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Replay doesn't notify registered notifiers but the one passed to it. So it's meaningless to check the registered notifier's synthetic flag. There is no issue currently as all replay use cases have MAP flag set, but let's be robust. Signed-off-by: Zhenzhong Duan Reviewed-by: Peter Xu --- hw/i386/intel_iommu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index 34af12f392f5..f046f8591335 100644 --- a/hw/i386/intel_iommu.c +++ b/hw/i386/intel_iommu.c @@ -3837,7 +3837,7 @@ static void vtd_iommu_replay(IOMMUMemoryRegion *iommu_mr, IOMMUNotifier *n) PCI_FUNC(vtd_as->devfn), vtd_get_domain_id(s, &ce, vtd_as->pasid), ce.hi, ce.lo); - if (vtd_as_has_map_notifier(vtd_as)) { + if (n->notifier_flags & IOMMU_NOTIFIER_MAP) { /* This is required only for MAP typed notifiers */ vtd_page_walk_info info = { .hook_fn = vtd_replay_hook, From patchwork Thu Jun 15 03:26:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Duan, Zhenzhong" X-Patchwork-Id: 13280672 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B082CEB64DC for ; Thu, 15 Jun 2023 03:39:41 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q9dpb-0005gB-VN; Wed, 14 Jun 2023 23:39:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q9dpa-0005fR-6Z for qemu-devel@nongnu.org; Wed, 14 Jun 2023 23:39:30 -0400 Received: from mga03.intel.com ([134.134.136.65]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q9dpY-0003qf-IA for qemu-devel@nongnu.org; Wed, 14 Jun 2023 23:39:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1686800368; x=1718336368; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RoxDv0+qZciLZzEjP7f5vJB8qDrFKIjClB6hYFbL37E=; b=IljdaFQdnsgjTXQplEAole8+Z36DkvlMMjPqAdgVbopRi50tl68ZxyAM jj9L5yrJ0jRoA6eNT4JN3fn6RMXsRrh7mRybhJYGG/kosUWJCnIphUfHj 2XUdtbNvlo3jouN8yQH9JJlIWehGs2tk0rn/KjEBlKCCmWA9Q3c1T15Vt xOlAyIX4BszQVQ+Sl3FG4jGEqYDvMX/GRPs0EG8d9itIjOrh4553/k0kX w3H9JFkGWzpWKHg3C3GeFt6W2okkEcQ+gYcCcy8Lvhe3fhfM2TPBb0FTs 5NJKDi9IUftnhdJ+ehxp7oTAnrFOk0sjtNuE5rrWD2oHynKW8tOSVvthJ A==; X-IronPort-AV: E=McAfee;i="6600,9927,10741"; a="362182819" X-IronPort-AV: E=Sophos;i="6.00,243,1681196400"; d="scan'208";a="362182819" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2023 20:39:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10741"; a="662626527" X-IronPort-AV: E=Sophos;i="6.00,243,1681196400"; d="scan'208";a="662626527" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2023 20:39:24 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: mst@redhat.com, peterx@redhat.com, jasowang@redhat.com, pbonzini@redhat.com, richard.henderson@linaro.org, eduardo@habkost.net, marcel.apfelbaum@gmail.com, alex.williamson@redhat.com, kwankhede@nvidia.com, cjia@nvidia.com, yi.l.liu@intel.com, chao.p.peng@intel.com Subject: [PATCH v4 3/3] intel_iommu: Fix address space unmap Date: Thu, 15 Jun 2023 11:26:26 +0800 Message-Id: <20230615032626.314476-4-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230615032626.314476-1-zhenzhong.duan@intel.com> References: <20230615032626.314476-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=134.134.136.65; envelope-from=zhenzhong.duan@intel.com; helo=mga03.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org During address space unmap, corresponding IOVA tree entries are also removed. But DMAMap is set beyond notifier's scope by 1, so in theory there is possibility to remove a continuous entry above the notifier's scope but falling in adjacent notifier's scope. There is no issue currently as no use cases allocate notifiers continuously, but let's be robust. Signed-off-by: Zhenzhong Duan Reviewed-by: Peter Xu --- hw/i386/intel_iommu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index f046f8591335..dcc334060cd6 100644 --- a/hw/i386/intel_iommu.c +++ b/hw/i386/intel_iommu.c @@ -3791,7 +3791,7 @@ static void vtd_address_space_unmap(VTDAddressSpace *as, IOMMUNotifier *n) n->start, size); map.iova = n->start; - map.size = size; + map.size = size - 1; /* Inclusive */ iova_tree_remove(as->iova_tree, map); }