From patchwork Fri Apr 29 09:25:16 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quan Xu X-Patchwork-Id: 8979121 Return-Path: X-Original-To: patchwork-xen-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 5BFA5BF29F for ; Fri, 29 Apr 2016 09:31:11 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 3354F200FF for ; Fri, 29 Apr 2016 09:31:10 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F3C9F20256 for ; Fri, 29 Apr 2016 09:31:08 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1aw4jP-0000Ly-1G; Fri, 29 Apr 2016 09:29:03 +0000 Received: from mail6.bemta6.messagelabs.com ([85.158.143.247]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1aw4jO-0000Hi-DV for xen-devel@lists.xen.org; Fri, 29 Apr 2016 09:29:02 +0000 Received: from [85.158.143.35] by server-1.bemta-6.messagelabs.com id 0F/74-18833-E5923275; Fri, 29 Apr 2016 09:29:02 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrFLMWRWlGSWpSXmKPExsVywNwkQjdWUzn cYMYTA4slHxezODB6HN39mymAMYo1My8pvyKBNWPd1GmsBdctKxZv2c3ewHhIr4uRg0NIoELi 4d+4LkZODgkBXokjy2awQtj+Eh//H2WHKKmRWPHNAyTMJqAoseHiciYQW0RAWuLa58uMXYxcH MwCC5gllvyYzQ6SEBaIk1jWv40ZxGYRUJVY0d/IAjKHV8BJYvUsMYjxChLLvqwFK+EECl+//p 4NxBYScJT40P2DdQIj7wJGhlWM6sWpRWWpRbpGeklFmekZJbmJmTm6hgZmermpxcWJ6ak5iUn Fesn5uZsYgWHAAAQ7GJf9dTrEKMnBpCTKG8KpHC7El5SfUpmRWJwRX1Sak1p8iFGGg0NJgjdO AygnWJSanlqRlpkDDEiYtAQHj5IIrxFImre4IDG3ODMdInWKUVFKnFcLJCEAksgozYNrg0XBJ UZZKWFeRqBDhHgKUotyM0tQ5V8xinMwKgnzdoBM4cnMK4Gb/gpoMRPQYoFNiiCLSxIRUlINjD oXplpb8YVtPvVaz+g/4+NnoQdZoqsWaa/4HL1L8nPkmtgZCVs73pjOX56+ySXguDDj1aCzJqF mury2l6c2Rlj3lubNVxHVjrt4pNErZHN/t36DSk3OZzvP81lHWDgk0g/fnLosdfn7nZt6Wf5M NPET6Xo2iz8wusZh9lqrD73J/k/+FuwoVWIpzkg01GIuKk4EAKiCABZ9AgAA X-Env-Sender: quan.xu@intel.com X-Msg-Ref: server-2.tower-21.messagelabs.com!1461922125!2123467!10 X-Originating-IP: [192.55.52.88] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTkyLjU1LjUyLjg4ID0+IDM3NDcyNQ==\n X-StarScan-Received: X-StarScan-Version: 8.34; banners=-,-,- X-VirusChecked: Checked Received: (qmail 45852 invoked from network); 29 Apr 2016 09:29:01 -0000 Received: from mga01.intel.com (HELO mga01.intel.com) (192.55.52.88) by server-2.tower-21.messagelabs.com with SMTP; 29 Apr 2016 09:29:01 -0000 Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga101.fm.intel.com with ESMTP; 29 Apr 2016 02:29:00 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.24,551,1455004800"; d="scan'208";a="965266506" Received: from xen-commits.sh.intel.com ([10.239.82.178]) by orsmga002.jf.intel.com with ESMTP; 29 Apr 2016 02:28:58 -0700 From: Quan Xu To: xen-devel@lists.xen.org Date: Fri, 29 Apr 2016 17:25:16 +0800 Message-Id: <1461921917-48394-10-git-send-email-quan.xu@intel.com> X-Mailer: git-send-email 1.8.3.2 In-Reply-To: <1461921917-48394-1-git-send-email-quan.xu@intel.com> References: <1461921917-48394-1-git-send-email-quan.xu@intel.com> Cc: Kevin Tian , Stefano Stabellini , Keir Fraser , Quan Xu , Liu Jinsong , dario.faggioli@citrix.com, Julien Grall , Jan Beulich , Andrew Cooper , Feng Wu , Suravee Suthikulpanit Subject: [Xen-devel] [PATCH v3 09/10] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Propagate the IOMMU Device-TLB flush error up to IOMMU suspending. Signed-off-by: Quan Xu CC: Jan Beulich CC: Liu Jinsong CC: Keir Fraser CC: Andrew Cooper CC: Suravee Suthikulpanit CC: Stefano Stabellini CC: Julien Grall CC: Kevin Tian CC: Feng Wu --- xen/arch/x86/acpi/power.c | 15 ++++++++++- xen/drivers/passthrough/amd/iommu_init.c | 9 ++++++- xen/drivers/passthrough/amd/pci_amd_iommu.c | 2 +- xen/drivers/passthrough/iommu.c | 7 +++-- xen/drivers/passthrough/vtd/iommu.c | 39 ++++++++++++++++++++------- xen/include/asm-x86/hvm/svm/amd-iommu-proto.h | 2 +- xen/include/xen/iommu.h | 4 +-- 7 files changed, 61 insertions(+), 17 deletions(-) diff --git a/xen/arch/x86/acpi/power.c b/xen/arch/x86/acpi/power.c index 2885e31..9097333 100644 --- a/xen/arch/x86/acpi/power.c +++ b/xen/arch/x86/acpi/power.c @@ -45,6 +45,8 @@ void do_suspend_lowlevel(void); static int device_power_down(void) { + int err; + console_suspend(); time_suspend(); @@ -53,11 +55,22 @@ static int device_power_down(void) ioapic_suspend(); - iommu_suspend(); + err = iommu_suspend(); + + if ( err ) + goto iommu_suspend_error; lapic_suspend(); return 0; + + iommu_suspend_error: + ioapic_resume(); + i8259A_resume(); + time_resume(); + console_resume(); + + return err; } static void device_power_up(void) diff --git a/xen/drivers/passthrough/amd/iommu_init.c b/xen/drivers/passthrough/amd/iommu_init.c index 4536106..02588aa 100644 --- a/xen/drivers/passthrough/amd/iommu_init.c +++ b/xen/drivers/passthrough/amd/iommu_init.c @@ -1339,12 +1339,19 @@ static void invalidate_all_devices(void) iterate_ivrs_mappings(_invalidate_all_devices); } -void amd_iommu_suspend(void) +int amd_iommu_suspend(void) { struct amd_iommu *iommu; for_each_amd_iommu ( iommu ) disable_iommu(iommu); + + return 0; +} + +void amd_iommu_crash_shutdown(void) +{ + amd_iommu_suspend(); } void amd_iommu_resume(void) diff --git a/xen/drivers/passthrough/amd/pci_amd_iommu.c b/xen/drivers/passthrough/amd/pci_amd_iommu.c index c8ee8dc..fb8e816 100644 --- a/xen/drivers/passthrough/amd/pci_amd_iommu.c +++ b/xen/drivers/passthrough/amd/pci_amd_iommu.c @@ -628,6 +628,6 @@ const struct iommu_ops amd_iommu_ops = { .suspend = amd_iommu_suspend, .resume = amd_iommu_resume, .share_p2m = amd_iommu_share_p2m, - .crash_shutdown = amd_iommu_suspend, + .crash_shutdown = amd_iommu_crash_shutdown, .dump_p2m_table = amd_dump_p2m_table, }; diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c index 839852f..c565c66 100644 --- a/xen/drivers/passthrough/iommu.c +++ b/xen/drivers/passthrough/iommu.c @@ -407,11 +407,14 @@ int iommu_do_domctl( return ret; } -void iommu_suspend() +int __must_check iommu_suspend() { const struct iommu_ops *ops = iommu_get_ops(); + if ( iommu_enabled ) - ops->suspend(); + return ops->suspend(); + + return 0; } void iommu_share_p2m_table(struct domain* d) diff --git a/xen/drivers/passthrough/vtd/iommu.c b/xen/drivers/passthrough/vtd/iommu.c index bf8c9c1..cf847ec 100644 --- a/xen/drivers/passthrough/vtd/iommu.c +++ b/xen/drivers/passthrough/vtd/iommu.c @@ -542,11 +542,12 @@ static int iommu_flush_iotlb_psi( return status; } -static void iommu_flush_all(void) +static int __must_check iommu_flush_all(void) { struct acpi_drhd_unit *drhd; struct iommu *iommu; int flush_dev_iotlb; + int rc = 0, ret; flush_all_cache(); for_each_drhd_unit ( drhd ) @@ -554,8 +555,13 @@ static void iommu_flush_all(void) iommu = drhd->iommu; iommu_flush_context_global(iommu, 0); flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0; - iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb); + ret = iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb); + + if ( ret ) + rc = ret; } + + return rc; } static int iommu_flush_iotlb(struct domain *d, unsigned long gfn, @@ -1272,7 +1278,9 @@ static void __hwdom_init intel_iommu_hwdom_init(struct domain *d) setup_hwdom_pci_devices(d, setup_hwdom_device); setup_hwdom_rmrr(d); - iommu_flush_all(); + if ( iommu_flush_all() ) + printk(XENLOG_WARNING VTDPREFIX + " intel_iommu_hwdom_init: IOMMU flush all failed.\n"); for_each_drhd_unit ( drhd ) { @@ -2134,8 +2142,8 @@ static int init_vtd_hw(void) return -EIO; } } - iommu_flush_all(); - return 0; + + return iommu_flush_all(); } static void __hwdom_init setup_hwdom_rmrr(struct domain *d) @@ -2424,16 +2432,25 @@ static int intel_iommu_group_id(u16 seg, u8 bus, u8 devfn) } static u32 iommu_state[MAX_IOMMUS][MAX_IOMMU_REGS]; -static void vtd_suspend(void) + +static int vtd_suspend(void) { struct acpi_drhd_unit *drhd; struct iommu *iommu; u32 i; + int rc; if ( !iommu_enabled ) - return; + return 0; - iommu_flush_all(); + rc = iommu_flush_all(); + + if ( rc ) + { + printk(XENLOG_WARNING VTDPREFIX + " vtd_suspend: IOMMU flush all failed.\n"); + return rc; + } for_each_drhd_unit ( drhd ) { @@ -2462,6 +2479,8 @@ static void vtd_suspend(void) if ( !iommu_intremap && iommu_qinval ) disable_qinval(iommu); } + + return 0; } static void vtd_crash_shutdown(void) @@ -2472,7 +2491,9 @@ static void vtd_crash_shutdown(void) if ( !iommu_enabled ) return; - iommu_flush_all(); + if ( iommu_flush_all() ) + printk(XENLOG_WARNING VTDPREFIX + " vtd_crash_shutdown: IOMMU flush all failed.\n"); for_each_drhd_unit ( drhd ) { diff --git a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h index 9c51172..f540fc8 100644 --- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h +++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h @@ -119,7 +119,7 @@ extern unsigned long *shared_intremap_inuse; /* power management support */ void amd_iommu_resume(void); -void amd_iommu_suspend(void); +int amd_iommu_suspend(void); void amd_iommu_crash_shutdown(void); /* guest iommu support */ diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h index fe52428..dfa52ef 100644 --- a/xen/include/xen/iommu.h +++ b/xen/include/xen/iommu.h @@ -157,7 +157,7 @@ struct iommu_ops { unsigned int (*read_apic_from_ire)(unsigned int apic, unsigned int reg); int (*setup_hpet_msi)(struct msi_desc *); #endif /* CONFIG_X86 */ - void (*suspend)(void); + int (*suspend)(void); void (*resume)(void); void (*share_p2m)(struct domain *d); void (*crash_shutdown)(void); @@ -167,7 +167,7 @@ struct iommu_ops { void (*dump_p2m_table)(struct domain *d); }; -void iommu_suspend(void); +int __must_check iommu_suspend(void); void iommu_resume(void); void iommu_crash_shutdown(void); int iommu_get_reserved_device_memory(iommu_grdm_t *, void *);