From patchwork Mon Mar 10 05:46:55 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gavin Shan X-Patchwork-Id: 3801321 Return-Path: X-Original-To: patchwork-kvm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 4E84ABF540 for ; Mon, 10 Mar 2014 05:47:45 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 6D04C20381 for ; Mon, 10 Mar 2014 05:47:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7457D20380 for ; Mon, 10 Mar 2014 05:47:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753226AbaCJFrW (ORCPT ); Mon, 10 Mar 2014 01:47:22 -0400 Received: from e36.co.us.ibm.com ([32.97.110.154]:37338 "EHLO e36.co.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753193AbaCJFrQ (ORCPT ); Mon, 10 Mar 2014 01:47:16 -0400 Received: from /spool/local by e36.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Sun, 9 Mar 2014 23:47:15 -0600 Received: from d03dlp01.boulder.ibm.com (9.17.202.177) by e36.co.us.ibm.com (192.168.1.136) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Sun, 9 Mar 2014 23:47:14 -0600 Received: from b03cxnp08026.gho.boulder.ibm.com (b03cxnp08026.gho.boulder.ibm.com [9.17.130.18]) by d03dlp01.boulder.ibm.com (Postfix) with ESMTP id D98651FF001A for ; Sun, 9 Mar 2014 23:47:13 -0600 (MDT) Received: from d03av01.boulder.ibm.com (d03av01.boulder.ibm.com [9.17.195.167]) by b03cxnp08026.gho.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id s2A5ki6H3997962 for ; Mon, 10 Mar 2014 06:46:44 +0100 Received: from d03av01.boulder.ibm.com (localhost [127.0.0.1]) by d03av01.boulder.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id s2A5lDmW027603 for ; Sun, 9 Mar 2014 23:47:13 -0600 Received: from shangw (shangw.cn.ibm.com [9.125.213.121]) by d03av01.boulder.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with SMTP id s2A5lAFx027555; Sun, 9 Mar 2014 23:47:11 -0600 Received: by shangw (Postfix, from userid 1000) id 486803004D4; Mon, 10 Mar 2014 13:46:59 +0800 (CST) From: Gavin Shan To: alex.williamson@redhat.com Cc: kvm@vger.kernel.org, benh@kernel.crashing.org, aik@ozlabs.ru, Gavin Shan Subject: [PATCH 3/3] drivers/vfio/pci: Fix MSIx message lost Date: Mon, 10 Mar 2014 13:46:55 +0800 Message-Id: <1394430415-15574-4-git-send-email-shangw@linux.vnet.ibm.com> X-Mailer: git-send-email 1.7.10.4 In-Reply-To: <1394430415-15574-1-git-send-email-shangw@linux.vnet.ibm.com> References: <1394430415-15574-1-git-send-email-shangw@linux.vnet.ibm.com> X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 14031005-3532-0000-0000-0000002F2E93 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The problem is specific to the case of BIST reset issued to IPR adapter on the guest side. The IPR driver calls pci_save_state(), issues BIST reset and then pci_restore_state(). The issued BIST cleans out MSIx table and pci_restore_state() doesn't restore the MSIx table as expected. Eventually, MSIx messages with all zeros are sent and causes EEH error on Power platform. The patch fixes it by writing MSIx message to MSIx table before reenabling the individual interrupts in the following path: qemu/hw/pci/msix.c::msix_table_mmio_write msix_handle_mask_update msix_fire_vector_notifier qemu/hw/misc/vfio.c::vfio_msix_vector_use vfio_msix_vector_do_use IOCTL Command VFIO_DEVICE_SET_IRQS to VFIO-PCI vfio/pci/vfio_pci.c::vfio_pci_ioctl vfio/pci/vfio_pci_intrs.c::vfio_pci_set_irqs_ioctl vfio_pci_set_msi_trigger vfio_msi_set_block vfio_msi_set_vector_signal Reported-by: Wen Xiong Signed-off-by: Gavin Shan --- drivers/vfio/pci/vfio_pci_intrs.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/drivers/vfio/pci/vfio_pci_intrs.c b/drivers/vfio/pci/vfio_pci_intrs.c index 2103576..83e0638 100644 --- a/drivers/vfio/pci/vfio_pci_intrs.c +++ b/drivers/vfio/pci/vfio_pci_intrs.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include @@ -517,6 +518,7 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_device *vdev, struct pci_dev *pdev = vdev->pdev; int irq = msix ? vdev->msix[vector].vector : pdev->irq + vector; char *name = msix ? "vfio-msix" : "vfio-msi"; + struct msi_msg msg; struct eventfd_ctx *trigger; int ret; @@ -544,6 +546,15 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_device *vdev, return PTR_ERR(trigger); } + /* We possiblly lose the MSI/MSIx message in some cases, one + * of which is pci_save_state(), BIST reset and pci_restore_state() + * for IPR adapter. The BIST reset cleans out MSIx table and we + * don't have chance to restore it. Here, we have the trick to + * restore it before enabling individual interrupts. For MSI messages, + * it's harmless to write them back. + */ + get_cached_msi_msg(irq, &msg); + write_msi_msg(irq, &msg); ret = request_irq(irq, vfio_msihandler, 0, vdev->ctx[vector].name, trigger); if (ret) {