From patchwork Fri May 6 08:54:39 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quan Xu X-Patchwork-Id: 9030441 Return-Path: X-Original-To: patchwork-xen-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 23FF8BF4C7 for ; Fri, 6 May 2016 09:00:57 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id E5BB62035D for ; Fri, 6 May 2016 09:00:55 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2FC3520392 for ; Fri, 6 May 2016 09:00:54 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1aybak-0004ZB-GB; Fri, 06 May 2016 08:58:34 +0000 Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1aybaj-0004XN-74 for xen-devel@lists.xen.org; Fri, 06 May 2016 08:58:33 +0000 Received: from [193.109.254.147] by server-3.bemta-14.messagelabs.com id 8D/99-03281-8BC5C275; Fri, 06 May 2016 08:58:32 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrJLMWRWlGSWpSXmKPExsXS1tYhorsjRif c4O8eZoslHxezODB6HN39mymAMYo1My8pvyKBNeP1TpWC9W4Vd39dZG5gXGvexcjJISRQIfGr 4RkziC0hwCtxZNkM1i5GDiDbX2LpymIQU0igRuLGsTyQCjYBRYkNF5czgdgiAtIS1z5fZuxi5 OJgFljALLHkx2x2kHphgTiJ1n+1ICaLgKrEly2xIOW8Ak4SN142sUAsUpBY9mUt2FJOoHjb1H Z2iGMcJX5M28Y0gZF3ASPDKkaN4tSistQiXUMjvaSizPSMktzEzBxdQ0MTvdzU4uLE9NScxKR iveT83E2MwCBgAIIdjGenOR9ilORgUhLlXfVPO1yILyk/pTIjsTgjvqg0J7X4EKMMB4eSBK8h MKiEBItS01Mr0jJzgOEIk5bg4FES4f0bDZTmLS5IzC3OTIdInWJUlBLn5QPpEwBJZJTmwbXBY uASo6yUMC8j0CFCPAWpRbmZJajyrxjFORiVhHm5QKbwZOaVwE1/BbSYCWjx+7maIItLEhFSUg 2MrHNjtL/ddZk3hf/cr+lflAPunprlJMTtWrP1edy152eML6yUC+VN5mllKza8tLw5UiL7+IE /Zyfd7zryYvKDinuJdQ6P57YfSG6sW931T05P1O/VnClaWdH83yYIR79dWXLN+vdZ9cfdhZWW 998Wbah96+K6YElv73XmmJ+ObWr2Ek+1t0V4K7EUZyQaajEXFScCANtlEup8AgAA X-Env-Sender: quan.xu@intel.com X-Msg-Ref: server-9.tower-27.messagelabs.com!1462525094!39896475!10 X-Originating-IP: [134.134.136.20] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTM0LjEzNC4xMzYuMjAgPT4gMzU1MzU4\n X-StarScan-Received: X-StarScan-Version: 8.34; banners=-,-,- X-VirusChecked: Checked Received: (qmail 46801 invoked from network); 6 May 2016 08:58:31 -0000 Received: from mga02.intel.com (HELO mga02.intel.com) (134.134.136.20) by server-9.tower-27.messagelabs.com with SMTP; 6 May 2016 08:58:31 -0000 Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga101.jf.intel.com with ESMTP; 06 May 2016 01:58:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.24,586,1455004800"; d="scan'208";a="969963870" Received: from xen-commits.sh.intel.com ([10.239.82.178]) by orsmga002.jf.intel.com with ESMTP; 06 May 2016 01:58:28 -0700 From: Quan Xu To: xen-devel@lists.xen.org Date: Fri, 6 May 2016 16:54:39 +0800 Message-Id: <1462524880-67205-10-git-send-email-quan.xu@intel.com> X-Mailer: git-send-email 1.8.3.2 In-Reply-To: <1462524880-67205-1-git-send-email-quan.xu@intel.com> References: <1462524880-67205-1-git-send-email-quan.xu@intel.com> Cc: Kevin Tian , Stefano Stabellini , Keir Fraser , Quan Xu , Liu Jinsong , dario.faggioli@citrix.com, Julien Grall , Jan Beulich , Andrew Cooper , Feng Wu , Suravee Suthikulpanit Subject: [Xen-devel] [PATCH v4 09/10] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Propagate the IOMMU Device-TLB flush error up to IOMMU suspending. Signed-off-by: Quan Xu CC: Jan Beulich CC: Liu Jinsong CC: Keir Fraser CC: Andrew Cooper CC: Suravee Suthikulpanit CC: Stefano Stabellini CC: Julien Grall CC: Kevin Tian CC: Feng Wu --- xen/arch/x86/acpi/power.c | 72 ++++++++++++++++++++------- xen/drivers/passthrough/amd/iommu_init.c | 9 +++- xen/drivers/passthrough/amd/pci_amd_iommu.c | 2 +- xen/drivers/passthrough/iommu.c | 7 ++- xen/drivers/passthrough/vtd/iommu.c | 39 +++++++++++---- xen/include/asm-x86/hvm/svm/amd-iommu-proto.h | 2 +- xen/include/xen/iommu.h | 4 +- 7 files changed, 101 insertions(+), 34 deletions(-) diff --git a/xen/arch/x86/acpi/power.c b/xen/arch/x86/acpi/power.c index 2885e31..1f7b8bf 100644 --- a/xen/arch/x86/acpi/power.c +++ b/xen/arch/x86/acpi/power.c @@ -43,36 +43,71 @@ struct acpi_sleep_info acpi_sinfo; void do_suspend_lowlevel(void); +enum dev_power_type +{ + /* Use for all of device power type */ + TYPE_ALL, + TYPE_CONSOLE, + TYPE_TIME, + TYPE_I8259A, + TYPE_IOAPIC, + TYPE_IOMMU, + TYPE_LAPIC, + /* Use for error */ + TYPE_UNKNOWN +}; + static int device_power_down(void) { - console_suspend(); + if ( console_suspend() ) + return TYPE_CONSOLE; - time_suspend(); + if ( time_suspend() ) + return TYPE_TIME; - i8259A_suspend(); + if ( i8259A_suspend() ) + return TYPE_I8259A; + /* ioapic_suspend should never fail */ ioapic_suspend(); - iommu_suspend(); + if ( iommu_suspend() ) + return TYPE_IOMMU; - lapic_suspend(); + if ( lapic_suspend() ) + return TYPE_LAPIC; return 0; } -static void device_power_up(void) +static void device_power_up(enum dev_power_type type) { - lapic_resume(); - - iommu_resume(); - - ioapic_resume(); - - i8259A_resume(); - - time_resume(); - - console_resume(); + switch ( type ) + { + case TYPE_ALL: + /* fall through */ + case TYPE_LAPIC: + lapic_resume(); + /* fall through */ + case TYPE_IOMMU: + iommu_resume(); + /* fall through */ + case TYPE_IOAPIC: + ioapic_resume(); + /* fall through */ + case TYPE_I8259A: + i8259A_resume(); + /* fall through */ + case TYPE_TIME: + time_resume(); + /* fall through */ + case TYPE_CONSOLE: + console_resume(); + break; + default: + BUG(); + break; + } } static void freeze_domains(void) @@ -169,6 +204,7 @@ static int enter_state(u32 state) { printk(XENLOG_ERR "Some devices failed to power down."); system_state = SYS_STATE_resume; + device_power_up(error); goto done; } @@ -196,7 +232,7 @@ static int enter_state(u32 state) write_cr4(cr4 & ~X86_CR4_MCE); write_efer(read_efer()); - device_power_up(); + device_power_up(TYPE_ALL); mcheck_init(&boot_cpu_data, 0); write_cr4(cr4); diff --git a/xen/drivers/passthrough/amd/iommu_init.c b/xen/drivers/passthrough/amd/iommu_init.c index 4536106..02588aa 100644 --- a/xen/drivers/passthrough/amd/iommu_init.c +++ b/xen/drivers/passthrough/amd/iommu_init.c @@ -1339,12 +1339,19 @@ static void invalidate_all_devices(void) iterate_ivrs_mappings(_invalidate_all_devices); } -void amd_iommu_suspend(void) +int amd_iommu_suspend(void) { struct amd_iommu *iommu; for_each_amd_iommu ( iommu ) disable_iommu(iommu); + + return 0; +} + +void amd_iommu_crash_shutdown(void) +{ + amd_iommu_suspend(); } void amd_iommu_resume(void) diff --git a/xen/drivers/passthrough/amd/pci_amd_iommu.c b/xen/drivers/passthrough/amd/pci_amd_iommu.c index 70b7475..ac40f63 100644 --- a/xen/drivers/passthrough/amd/pci_amd_iommu.c +++ b/xen/drivers/passthrough/amd/pci_amd_iommu.c @@ -626,6 +626,6 @@ const struct iommu_ops amd_iommu_ops = { .suspend = amd_iommu_suspend, .resume = amd_iommu_resume, .share_p2m = amd_iommu_share_p2m, - .crash_shutdown = amd_iommu_suspend, + .crash_shutdown = amd_iommu_crash_shutdown, .dump_p2m_table = amd_dump_p2m_table, }; diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c index c013d91..8b26b1c 100644 --- a/xen/drivers/passthrough/iommu.c +++ b/xen/drivers/passthrough/iommu.c @@ -404,11 +404,14 @@ int iommu_do_domctl( return ret; } -void iommu_suspend() +int __must_check iommu_suspend() { const struct iommu_ops *ops = iommu_get_ops(); + if ( iommu_enabled ) - ops->suspend(); + return ops->suspend(); + + return 0; } void iommu_share_p2m_table(struct domain* d) diff --git a/xen/drivers/passthrough/vtd/iommu.c b/xen/drivers/passthrough/vtd/iommu.c index bf77417..29cf870 100644 --- a/xen/drivers/passthrough/vtd/iommu.c +++ b/xen/drivers/passthrough/vtd/iommu.c @@ -541,11 +541,12 @@ static int iommu_flush_iotlb_psi( return status; } -static void iommu_flush_all(void) +static int __must_check iommu_flush_all(void) { struct acpi_drhd_unit *drhd; struct iommu *iommu; int flush_dev_iotlb; + int rc = 0, ret; flush_all_cache(); for_each_drhd_unit ( drhd ) @@ -553,8 +554,13 @@ static void iommu_flush_all(void) iommu = drhd->iommu; iommu_flush_context_global(iommu, 0); flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0; - iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb); + ret = iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb); + + if ( ret ) + rc = ret; } + + return rc; } static int iommu_flush_iotlb(struct domain *d, unsigned long gfn, @@ -1267,7 +1273,9 @@ static void __hwdom_init intel_iommu_hwdom_init(struct domain *d) setup_hwdom_pci_devices(d, setup_hwdom_device); setup_hwdom_rmrr(d); - iommu_flush_all(); + if ( iommu_flush_all() ) + printk(XENLOG_WARNING VTDPREFIX + " intel_iommu_hwdom_init: IOMMU flush all failed.\n"); for_each_drhd_unit ( drhd ) { @@ -2126,8 +2134,8 @@ static int init_vtd_hw(void) return -EIO; } } - iommu_flush_all(); - return 0; + + return iommu_flush_all(); } static void __hwdom_init setup_hwdom_rmrr(struct domain *d) @@ -2416,16 +2424,25 @@ static int intel_iommu_group_id(u16 seg, u8 bus, u8 devfn) } static u32 iommu_state[MAX_IOMMUS][MAX_IOMMU_REGS]; -static void vtd_suspend(void) + +static int vtd_suspend(void) { struct acpi_drhd_unit *drhd; struct iommu *iommu; u32 i; + int rc; if ( !iommu_enabled ) - return; + return 0; - iommu_flush_all(); + rc = iommu_flush_all(); + + if ( rc ) + { + printk(XENLOG_WARNING VTDPREFIX + " vtd_suspend: IOMMU flush all failed.\n"); + return rc; + } for_each_drhd_unit ( drhd ) { @@ -2454,6 +2471,8 @@ static void vtd_suspend(void) if ( !iommu_intremap && iommu_qinval ) disable_qinval(iommu); } + + return 0; } static void vtd_crash_shutdown(void) @@ -2464,7 +2483,9 @@ static void vtd_crash_shutdown(void) if ( !iommu_enabled ) return; - iommu_flush_all(); + if ( iommu_flush_all() ) + printk(XENLOG_WARNING VTDPREFIX + " vtd_crash_shutdown: IOMMU flush all failed.\n"); for_each_drhd_unit ( drhd ) { diff --git a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h index 9c51172..f540fc8 100644 --- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h +++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h @@ -119,7 +119,7 @@ extern unsigned long *shared_intremap_inuse; /* power management support */ void amd_iommu_resume(void); -void amd_iommu_suspend(void); +int amd_iommu_suspend(void); void amd_iommu_crash_shutdown(void); /* guest iommu support */ diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h index 6a64014..fee5a8f 100644 --- a/xen/include/xen/iommu.h +++ b/xen/include/xen/iommu.h @@ -175,7 +175,7 @@ struct iommu_ops { unsigned int (*read_apic_from_ire)(unsigned int apic, unsigned int reg); int (*setup_hpet_msi)(struct msi_desc *); #endif /* CONFIG_X86 */ - void (*suspend)(void); + int (*suspend)(void); void (*resume)(void); void (*share_p2m)(struct domain *d); void (*crash_shutdown)(void); @@ -185,7 +185,7 @@ struct iommu_ops { void (*dump_p2m_table)(struct domain *d); }; -void iommu_suspend(void); +int __must_check iommu_suspend(void); void iommu_resume(void); void iommu_crash_shutdown(void); int iommu_get_reserved_device_memory(iommu_grdm_t *, void *);