From patchwork Mon Apr 8 01:27:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leo Yan X-Patchwork-Id: 10888709 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ECFED139A for ; Mon, 8 Apr 2019 01:27:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D56A120748 for ; Mon, 8 Apr 2019 01:27:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C9FBA2864E; Mon, 8 Apr 2019 01:27:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4ACAE28613 for ; Mon, 8 Apr 2019 01:27:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726513AbfDHB16 (ORCPT ); Sun, 7 Apr 2019 21:27:58 -0400 Received: from mail-yw1-f67.google.com ([209.85.161.67]:44535 "EHLO mail-yw1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726527AbfDHB16 (ORCPT ); Sun, 7 Apr 2019 21:27:58 -0400 Received: by mail-yw1-f67.google.com with SMTP id c4so4352716ywa.11 for ; Sun, 07 Apr 2019 18:27:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9jMj4zisIi2PTAUSf9rNiwxCUVuPxEzAObjIhEFNUsQ=; b=uVhjfmZH5BN7AtgpBxgCNuaXgvIxp2FyOJ5KAtnLsAQKmtND0SiVJIF2W29FSsgj+C +H3IzdtwndJik/uDfjKnUyVkTyENTbeYN2oIzULB9woIWZBMIQ9xJDUzHjPnkMXfht9m 5+Nz3tTvx5owKr0OsXkL7XlJbECEzk9MpQ2vaEGPhptYPAyibbv/pjAYhN9XC5tNUz79 cc3k0aWWE39sr+VhAKpM76d9sh3S1+VqBd70N0oBq5JXS6k/kbg+8LvlsiY7Q4jO7NEh hvtMEkIOFOWUDh8NrEHNpET0VXD3IDODIN1w5FL/F9casKED/D6Oianoo2kzvxghaNU3 gdPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9jMj4zisIi2PTAUSf9rNiwxCUVuPxEzAObjIhEFNUsQ=; b=O5fLjf3UYD7peXfZfGnFgtqFE5dTFHIh0XjPTAyta9Fe/cAFXuI9rRGvyGFxu9s8T1 Luwm+zdvmQgZTKLj8qhuhRx/UsmsoP0IeXcfngRnUEirhk+QvlvH2oKIFkPthc3kNm/x qgTsmiotUuT4hl54N86Jsc4IOejIxU08ZpxioZ7R1SzHsnEgCaOXAak0wCfaK+aNqa2k vz5tthPF9+BuUblPFYFD1yTzd/3ITVErbMma3vJGjj1tsQmA6LVXhi8XtHJylEgBmrXl 4fqmoiCNe7Zb7ksedXDnin1Vo+hA2QTSMWDgBYTK2pFrvEkKhijJnw0KF2NM/eAwPEWH M/2A== X-Gm-Message-State: APjAAAVl83O/offT9KTRFkIBWe8L3IFKjtqWjCPWcS/DaeZqXutsUBvb vA+f8uD3Yas0DNh3eQlOAMwcfBqF1f81mYMb X-Google-Smtp-Source: APXvYqyt8WJ2bAhRgvq3AHr/y7EBERk4LqRyLa1Gh4nLELo8/NnIUle1muXyjkwAwMcCQrN5ns0MbA== X-Received: by 2002:a81:2e83:: with SMTP id u125mr20760789ywu.18.1554686876734; Sun, 07 Apr 2019 18:27:56 -0700 (PDT) Received: from localhost.localdomain (li931-65.members.linode.com. [45.56.113.65]) by smtp.gmail.com with ESMTPSA id 190sm10494919ywd.62.2019.04.07.18.27.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 07 Apr 2019 18:27:55 -0700 (PDT) From: Leo Yan To: "kvm@vger.kernel.org" , "kvmarm@lists.cs.columbia.edu" , Will Deacon , Jean-Philippe Brucker , Marc Zyngier , Eric Auger , Robin Murphy Cc: Leo Yan Subject: [PATCH v4 3/3] vfio-pci: Re-enable INTx mode when disable MSI/MSIX Date: Mon, 8 Apr 2019 09:27:19 +0800 Message-Id: <20190408012719.16158-4-leo.yan@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190408012719.16158-1-leo.yan@linaro.org> References: <20190408012719.16158-1-leo.yan@linaro.org> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Since PCI forbids enabling INTx, MSI or MSIX at the same time, it's by default to disable INTx mode when enable MSI/MSIX mode; but this logic is easily broken if the guest PCI driver detects the MSI/MSIX cannot work as expected and tries to rollback to use INTx mode. In this case, the INTx mode has been disabled and has no chance to re-enable it, thus both INTx mode and MSI/MSIX mode cannot work in vfio. Below shows the detailed flow for introducing this issue: vfio_pci_configure_dev_irqs() `-> vfio_pci_enable_intx() vfio_pci_enable_msis() `-> vfio_pci_disable_intx() vfio_pci_disable_msis() => Guest PCI driver disables MSI To fix this issue, when disable MSI/MSIX we need to check if INTx mode is available for this device or not; if the device can support INTx then re-enable it so that the device can fallback to use it. Since vfio_pci_disable_intx() / vfio_pci_enable_intx() pair functions may be called for multiple times, this patch uses 'intx_fd == -1' to denote the INTx is disabled, the pair functions can directly bail out when detect INTx has been disabled and enabled respectively. Suggested-by: Jean-Philippe Brucker Signed-off-by: Leo Yan Reviewed-by: Jean-Philippe Brucker --- vfio/pci.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/vfio/pci.c b/vfio/pci.c index 3c39844..f17498e 100644 --- a/vfio/pci.c +++ b/vfio/pci.c @@ -28,6 +28,7 @@ struct vfio_irq_eventfd { msi_update_state(state, val, VFIO_PCI_MSI_STATE_EMPTY) static void vfio_pci_disable_intx(struct kvm *kvm, struct vfio_device *vdev); +static int vfio_pci_enable_intx(struct kvm *kvm, struct vfio_device *vdev); static int vfio_pci_enable_msis(struct kvm *kvm, struct vfio_device *vdev, bool msix) @@ -50,17 +51,14 @@ static int vfio_pci_enable_msis(struct kvm *kvm, struct vfio_device *vdev, if (!msi_is_enabled(msis->virt_state)) return 0; - if (pdev->irq_modes & VFIO_PCI_IRQ_MODE_INTX) { + if (pdev->irq_modes & VFIO_PCI_IRQ_MODE_INTX) /* * PCI (and VFIO) forbids enabling INTx, MSI or MSIX at the same * time. Since INTx has to be enabled from the start (we don't - * have a reliable way to know when the user starts using it), + * have a reliable way to know when the guest starts using it), * disable it now. */ vfio_pci_disable_intx(kvm, vdev); - /* Permanently disable INTx */ - pdev->irq_modes &= ~VFIO_PCI_IRQ_MODE_INTX; - } eventfds = (void *)msis->irq_set + sizeof(struct vfio_irq_set); @@ -162,7 +160,16 @@ static int vfio_pci_disable_msis(struct kvm *kvm, struct vfio_device *vdev, msi_set_enabled(msis->phys_state, false); msi_set_empty(msis->phys_state, true); - return 0; + /* + * When MSI or MSIX is disabled, this might be called when + * PCI driver detects the MSI interrupt failure and wants to + * rollback to INTx mode. Thus enable INTx if the device + * supports INTx mode in this case. + */ + if (pdev->irq_modes & VFIO_PCI_IRQ_MODE_INTX) + ret = vfio_pci_enable_intx(kvm, vdev); + + return ret >= 0 ? 0 : ret; } static int vfio_pci_update_msi_entry(struct kvm *kvm, struct vfio_device *vdev, @@ -1002,6 +1009,9 @@ static void vfio_pci_disable_intx(struct kvm *kvm, struct vfio_device *vdev) .index = VFIO_PCI_INTX_IRQ_INDEX, }; + if (pdev->intx_fd == -1) + return; + pr_debug("user requested MSI, disabling INTx %d", gsi); ioctl(vdev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); @@ -1009,6 +1019,7 @@ static void vfio_pci_disable_intx(struct kvm *kvm, struct vfio_device *vdev) close(pdev->intx_fd); close(pdev->unmask_fd); + pdev->intx_fd = -1; } static int vfio_pci_enable_intx(struct kvm *kvm, struct vfio_device *vdev) @@ -1020,6 +1031,9 @@ static int vfio_pci_enable_intx(struct kvm *kvm, struct vfio_device *vdev) struct vfio_pci_device *pdev = &vdev->pci; int gsi = pdev->intx_gsi; + if (pdev->intx_fd != -1) + return 0; + /* * PCI IRQ is level-triggered, so we use two eventfds. trigger_fd * signals an interrupt from host to guest, and unmask_fd signals the @@ -1122,6 +1136,8 @@ static int vfio_pci_init_intx(struct kvm *kvm, struct vfio_device *vdev) /* Guest is going to ovewrite our irq_line... */ pdev->intx_gsi = pdev->hdr.irq_line - KVM_IRQ_OFFSET; + pdev->intx_fd = -1; + return 0; }