From patchwork Thu Sep 15 06:11:48 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gibson X-Patchwork-Id: 9332831 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A44A06077F for ; Thu, 15 Sep 2016 06:13:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 938A629288 for ; Thu, 15 Sep 2016 06:13:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 85CC329298; Thu, 15 Sep 2016 06:13:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7710E29288 for ; Thu, 15 Sep 2016 06:13:15 +0000 (UTC) Received: from localhost ([::1]:60044 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bkPv8-0002Yi-24 for patchwork-qemu-devel@patchwork.kernel.org; Thu, 15 Sep 2016 02:13:14 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46653) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bkPuo-0002Y2-1q for qemu-devel@nongnu.org; Thu, 15 Sep 2016 02:12:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bkPui-0008Ua-Rk for qemu-devel@nongnu.org; Thu, 15 Sep 2016 02:12:52 -0400 Received: from ozlabs.org ([103.22.144.67]:52550) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bkPui-0008Mi-Ea; Thu, 15 Sep 2016 02:12:48 -0400 Received: by ozlabs.org (Postfix, from userid 1007) id 3sZSjh10pGz9s65; Thu, 15 Sep 2016 16:11:52 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gibson.dropbear.id.au; s=201602; t=1473919912; bh=tx+9ksnkdidt/+Om66jWFggR9AunWfV1PgRvdOkVJ+k=; h=From:To:Cc:Subject:Date:From; b=KODn22/Eq23onvNmcRwqtBlgXJu8VQfhz6o2RqwERh8lnVGDFWoBbN50XRIU8JPQJ obY9Ljx18K9W73P3+OG6VDAQIBsE2f4OlYvO7yL1se/vE/39CJXgxcWQccXGjplIK/ qG2Npin9eKw36nsIM6HzA0PAHEcWlNBp/PwpP80w= From: David Gibson To: alex.williamson@redhat.com Date: Thu, 15 Sep 2016 16:11:48 +1000 Message-Id: <1473919908-20653-1-git-send-email-david@gibson.dropbear.id.au> X-Mailer: git-send-email 2.7.4 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 103.22.144.67 Subject: [Qemu-devel] [PATCH] vfio: Fix regression in MSI routing configuration X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: gwshan@au1.ibm.com, qemu-ppc@nongnu.org, qemu-devel@nongnu.org, peterx@redhat.com, David Gibson Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP d1f6af6 "kvm-irqchip: simplify kvm_irqchip_add_msi_route" was a cleanup of kvmchip routing configuration, that was mostly intended for x86. However, it also contains a subtle change in behaviour which breaks EEH[1] error recovery on certain VFIO passthrough devices on spapr guests. So far it's only been seen on a BCM5719 NIC on a POWER8 server, but there may be other hardware with the same problem. It's also possible there could be circumstances where it causes a bug on x86 as well, though I don't know of any obvious candidates. Prior to d1f6af6, both vfio_msix_vector_do_use() and vfio_add_kvm_msi_virq() used msg == NULL as a special flag to mark this as the "dummy" vector used to make the host hardware state sync with the guest expected hardware state in terms of MSI configuration. Specifically that flag caused vfio_add_kvm_msi_virq() to become a no-op, meaning the dummy irq would always be delivered via qemu. d1f6af6 changed vfio_add_kvm_msi_virq() so it takes a vector number instead of the msg parameter, and determines the correct message itself. The test for !msg was removed, and not replaced with anything there or in the caller. With an spapr guest which has a VFIO device, if an EEH error occurs on the host hardware, then the device will be isolated then reset. This is a combination of host and guest action, mediated by some EEH related hypercalls. I haven't fully traced the mechanics, but somehow installing the kvm irqchip route for the dummy irq on the BCM5719 means that after EEH reset and recovery, at least some irqs are no longer delivered to the guest. In particular, the guest never gets the link up event, and so the NIC is effectively dead. [1] EEH (Enhanced Error Handling) is an IBM POWER server specific PCI-* error reporting and recovery mechanism. The concept is somewhat similar to PCI-E AER, but the details are different. Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1373802 Cc: Alex Williamson Cc: Peter Xu Cc: Gavin Shan Signed-off-by: David Gibson --- hw/vfio/pci.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 7bfa17c..a5a620a 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -496,7 +496,9 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, vfio_update_kvm_msi_virq(vector, *msg, pdev); } } else { - vfio_add_kvm_msi_virq(vdev, vector, nr, true); + if (msg) { + vfio_add_kvm_msi_virq(vdev, vector, nr, true); + } } /*