From patchwork Wed Jun 8 08:59:00 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quan Xu X-Patchwork-Id: 9163735 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 535B060467 for ; Wed, 8 Jun 2016 09:05:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4317E28345 for ; Wed, 8 Jun 2016 09:05:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 37FDA28363; Wed, 8 Jun 2016 09:05:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 8B48C28392 for ; Wed, 8 Jun 2016 09:05:38 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bAZOm-0004A5-37; Wed, 08 Jun 2016 09:03:40 +0000 Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bAZOl-00049G-Fp for xen-devel@lists.xen.org; Wed, 08 Jun 2016 09:03:39 +0000 Received: from [193.109.254.147] by server-14.bemta-14.messagelabs.com id 1B/03-05219-A6FD7575; Wed, 08 Jun 2016 09:03:38 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrFLMWRWlGSWpSXmKPExsVywNwkQjfrfni 4QdcvFoslHxezODB6HN39mymAMYo1My8pvyKBNePankWMBSvdKxpn3mVpYNxj0cXIySEkUClx YcFsNhBbQoBX4siyGawQtr/E9Fmz2bsYOYBqaiSeTgoHCbMJqEjMaH7HDmKLCEhLXPt8mbGLk YuDWWABs8SSHxD1wgL5EnM/uYHUsAioSrxvOQA2nlfAUWLJjE8sEOMVJJZ9WcsMUs4p4CSx9V gpxDWOEtt2P2GZwMi7gJFhFaN6cWpRWWqRrpleUlFmekZJbmJmjq6hoYlebmpxcWJ6ak5iUrF ecn7uJkZgGDAAwQ7GvxOcDzFKcjApifIquoeHC/El5adUZiQWZ8QXleakFh9ilOHgUJLgrb0H lBMsSk1PrUjLzAEGJExagoNHSYRXCiTNW1yQmFucmQ6ROsWoKCXOKwqSEABJZJTmwbXBouASo 6yUMC8j0CFCPAWpRbmZJajyrxjFORiVhHnDQKbwZOaVwE1/BbSYCWjx8iNgi0sSEVJSDYyJCX ePG75u1lh09MxRoajE/N/n7x9KYDg3rW77q4tiZZoZqh55p+ILfn1SlCtOV0xa/CjDbd0Vw7e t1Q8m280sUP1dceNbtodFXsSbD4lGWTb7rWz4dI7tm5GRY/lupozfuf4NS8Krtm3rXsVjGi25 g8Houp53YYapRIdGr8TTujpPfZGsKCWW4oxEQy3mouJEAOAJD3x9AgAA X-Env-Sender: quan.xu@intel.com X-Msg-Ref: server-9.tower-27.messagelabs.com!1465376607!46437496!6 X-Originating-IP: [192.55.52.88] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTkyLjU1LjUyLjg4ID0+IDM3NDcyNQ==\n X-StarScan-Received: X-StarScan-Version: 8.46; banners=-,-,- X-VirusChecked: Checked Received: (qmail 29188 invoked from network); 8 Jun 2016 09:03:37 -0000 Received: from mga01.intel.com (HELO mga01.intel.com) (192.55.52.88) by server-9.tower-27.messagelabs.com with SMTP; 8 Jun 2016 09:03:37 -0000 Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga101.fm.intel.com with ESMTP; 08 Jun 2016 02:03:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.26,438,1459839600"; d="scan'208";a="983223969" Received: from xen-commits.sh.intel.com ([10.239.82.178]) by fmsmga001.fm.intel.com with ESMTP; 08 Jun 2016 02:03:35 -0700 From: "Xu, Quan" To: xen-devel@lists.xen.org Date: Wed, 8 Jun 2016 16:59:00 +0800 Message-Id: <1465376344-28290-8-git-send-email-quan.xu@intel.com> X-Mailer: git-send-email 1.8.3.2 In-Reply-To: <1465376344-28290-1-git-send-email-quan.xu@intel.com> References: <1465376344-28290-1-git-send-email-quan.xu@intel.com> Cc: Kevin Tian , Stefano Stabellini , Keir Fraser , Quan Xu , Liu Jinsong , dario.faggioli@citrix.com, Julien Grall , Jan Beulich , Andrew Cooper , Feng Wu , Suravee Suthikulpanit Subject: [Xen-devel] [PATCH v7 07/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending (top level ones) X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Quan Xu Signed-off-by: Quan Xu CC: Jan Beulich CC: Liu Jinsong CC: Keir Fraser CC: Andrew Cooper CC: Suravee Suthikulpanit CC: Stefano Stabellini CC: Julien Grall CC: Kevin Tian CC: Feng Wu v7: 1. return SAVED_ALL at the bottom of device_power_down(), instead of SAVED_NONE. 2. drop the 'if ( error > 0 )', calling device_power_up(error) without any if(). 3. for vtd_suspend(): - drop pointless initializer. - return 0 at the bottom to make obvious that no error path comes there. Acked-by: Suravee Suthikulpanit Acked-by: Kevin Tian --- xen/arch/x86/acpi/power.c | 73 ++++++++++++++++++++------- xen/drivers/passthrough/amd/iommu_init.c | 9 +++- xen/drivers/passthrough/amd/pci_amd_iommu.c | 2 +- xen/drivers/passthrough/iommu.c | 6 ++- xen/drivers/passthrough/vtd/iommu.c | 35 +++++++++---- xen/include/asm-x86/hvm/svm/amd-iommu-proto.h | 2 +- xen/include/xen/iommu.h | 4 +- 7 files changed, 96 insertions(+), 35 deletions(-) diff --git a/xen/arch/x86/acpi/power.c b/xen/arch/x86/acpi/power.c index 2885e31..717a809 100644 --- a/xen/arch/x86/acpi/power.c +++ b/xen/arch/x86/acpi/power.c @@ -43,36 +43,70 @@ struct acpi_sleep_info acpi_sinfo; void do_suspend_lowlevel(void); +enum dev_power_saved +{ + SAVED_NONE, + SAVED_CONSOLE, + SAVED_TIME, + SAVED_I8259A, + SAVED_IOAPIC, + SAVED_IOMMU, + SAVED_LAPIC, + SAVED_ALL, +}; + static int device_power_down(void) { - console_suspend(); + if ( console_suspend() ) + return SAVED_NONE; - time_suspend(); + if ( time_suspend() ) + return SAVED_CONSOLE; - i8259A_suspend(); + if ( i8259A_suspend() ) + return SAVED_TIME; + /* ioapic_suspend cannot fail */ ioapic_suspend(); - iommu_suspend(); + if ( iommu_suspend() ) + return SAVED_IOAPIC; - lapic_suspend(); + if ( lapic_suspend() ) + return SAVED_IOMMU; - return 0; + return SAVED_ALL; } -static void device_power_up(void) +static void device_power_up(enum dev_power_saved saved) { - lapic_resume(); - - iommu_resume(); - - ioapic_resume(); - - i8259A_resume(); - - time_resume(); - - console_resume(); + switch ( saved ) + { + case SAVED_ALL: + case SAVED_LAPIC: + lapic_resume(); + /* fall through */ + case SAVED_IOMMU: + iommu_resume(); + /* fall through */ + case SAVED_IOAPIC: + ioapic_resume(); + /* fall through */ + case SAVED_I8259A: + i8259A_resume(); + /* fall through */ + case SAVED_TIME: + time_resume(); + /* fall through */ + case SAVED_CONSOLE: + console_resume(); + /* fall through */ + case SAVED_NONE: + break; + default: + BUG(); + break; + } } static void freeze_domains(void) @@ -169,6 +203,7 @@ static int enter_state(u32 state) { printk(XENLOG_ERR "Some devices failed to power down."); system_state = SYS_STATE_resume; + device_power_up(error); goto done; } @@ -196,7 +231,7 @@ static int enter_state(u32 state) write_cr4(cr4 & ~X86_CR4_MCE); write_efer(read_efer()); - device_power_up(); + device_power_up(SAVED_ALL); mcheck_init(&boot_cpu_data, 0); write_cr4(cr4); diff --git a/xen/drivers/passthrough/amd/iommu_init.c b/xen/drivers/passthrough/amd/iommu_init.c index 4536106..0b68596 100644 --- a/xen/drivers/passthrough/amd/iommu_init.c +++ b/xen/drivers/passthrough/amd/iommu_init.c @@ -1339,7 +1339,14 @@ static void invalidate_all_devices(void) iterate_ivrs_mappings(_invalidate_all_devices); } -void amd_iommu_suspend(void) +int amd_iommu_suspend(void) +{ + amd_iommu_crash_shutdown(); + + return 0; +} + +void amd_iommu_crash_shutdown(void) { struct amd_iommu *iommu; diff --git a/xen/drivers/passthrough/amd/pci_amd_iommu.c b/xen/drivers/passthrough/amd/pci_amd_iommu.c index 4a860af..7761241 100644 --- a/xen/drivers/passthrough/amd/pci_amd_iommu.c +++ b/xen/drivers/passthrough/amd/pci_amd_iommu.c @@ -633,6 +633,6 @@ const struct iommu_ops amd_iommu_ops = { .suspend = amd_iommu_suspend, .resume = amd_iommu_resume, .share_p2m = amd_iommu_share_p2m, - .crash_shutdown = amd_iommu_suspend, + .crash_shutdown = amd_iommu_crash_shutdown, .dump_p2m_table = amd_dump_p2m_table, }; diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c index 3a73fab..a9898fc 100644 --- a/xen/drivers/passthrough/iommu.c +++ b/xen/drivers/passthrough/iommu.c @@ -379,10 +379,12 @@ int __init iommu_setup(void) return rc; } -void iommu_suspend() +int iommu_suspend() { if ( iommu_enabled ) - iommu_get_ops()->suspend(); + return iommu_get_ops()->suspend(); + + return 0; } void iommu_resume() diff --git a/xen/drivers/passthrough/vtd/iommu.c b/xen/drivers/passthrough/vtd/iommu.c index 5366267..0f17afb 100644 --- a/xen/drivers/passthrough/vtd/iommu.c +++ b/xen/drivers/passthrough/vtd/iommu.c @@ -541,7 +541,7 @@ static int iommu_flush_iotlb_psi( return status; } -static void iommu_flush_all(void) +static int __must_check iommu_flush_all(void) { struct acpi_drhd_unit *drhd; struct iommu *iommu; @@ -555,6 +555,8 @@ static void iommu_flush_all(void) flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0; iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb); } + + return 0; } static void __intel_iommu_iotlb_flush(struct domain *d, unsigned long gfn, @@ -1259,7 +1261,9 @@ static void __hwdom_init intel_iommu_hwdom_init(struct domain *d) setup_hwdom_pci_devices(d, setup_hwdom_device); setup_hwdom_rmrr(d); - iommu_flush_all(); + if ( iommu_flush_all() ) + printk(XENLOG_WARNING VTDPREFIX + " IOMMU flush all failed for hardware domain\n"); for_each_drhd_unit ( drhd ) { @@ -2001,7 +2005,7 @@ int adjust_vtd_irq_affinities(void) } __initcall(adjust_vtd_irq_affinities); -static int init_vtd_hw(void) +static int __must_check init_vtd_hw(void) { struct acpi_drhd_unit *drhd; struct iommu *iommu; @@ -2099,8 +2103,8 @@ static int init_vtd_hw(void) return -EIO; } } - iommu_flush_all(); - return 0; + + return iommu_flush_all(); } static void __hwdom_init setup_hwdom_rmrr(struct domain *d) @@ -2389,16 +2393,25 @@ static int intel_iommu_group_id(u16 seg, u8 bus, u8 devfn) } static u32 iommu_state[MAX_IOMMUS][MAX_IOMMU_REGS]; -static void vtd_suspend(void) + +static int __must_check vtd_suspend(void) { struct acpi_drhd_unit *drhd; struct iommu *iommu; u32 i; + int rc; if ( !iommu_enabled ) - return; + return 0; - iommu_flush_all(); + rc = iommu_flush_all(); + if ( unlikely(rc) ) + { + printk(XENLOG_WARNING VTDPREFIX + " suspend: IOMMU flush all failed: %d\n", rc); + + return rc; + } for_each_drhd_unit ( drhd ) { @@ -2427,6 +2440,8 @@ static void vtd_suspend(void) if ( !iommu_intremap && iommu_qinval ) disable_qinval(iommu); } + + return 0; } static void vtd_crash_shutdown(void) @@ -2437,7 +2452,9 @@ static void vtd_crash_shutdown(void) if ( !iommu_enabled ) return; - iommu_flush_all(); + if ( iommu_flush_all() ) + printk(XENLOG_WARNING VTDPREFIX + " crash shutdown: IOMMU flush all failed\n"); for_each_drhd_unit ( drhd ) { diff --git a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h index ac9f036..d08dc0b 100644 --- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h +++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h @@ -119,7 +119,7 @@ extern unsigned long *shared_intremap_inuse; /* power management support */ void amd_iommu_resume(void); -void amd_iommu_suspend(void); +int __must_check amd_iommu_suspend(void); void amd_iommu_crash_shutdown(void); /* guest iommu support */ diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h index 57c9fbc..6535937 100644 --- a/xen/include/xen/iommu.h +++ b/xen/include/xen/iommu.h @@ -175,7 +175,7 @@ struct iommu_ops { unsigned int (*read_apic_from_ire)(unsigned int apic, unsigned int reg); int (*setup_hpet_msi)(struct msi_desc *); #endif /* CONFIG_X86 */ - void (*suspend)(void); + int __must_check (*suspend)(void); void (*resume)(void); void (*share_p2m)(struct domain *d); void (*crash_shutdown)(void); @@ -185,7 +185,7 @@ struct iommu_ops { void (*dump_p2m_table)(struct domain *d); }; -void iommu_suspend(void); +int __must_check iommu_suspend(void); void iommu_resume(void); void iommu_crash_shutdown(void); int iommu_get_reserved_device_memory(iommu_grdm_t *, void *);