From patchwork Wed Mar 20 06:20:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leo Yan X-Patchwork-Id: 10860879 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6463F139A for ; Wed, 20 Mar 2019 06:21:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 49C6C299D3 for ; Wed, 20 Mar 2019 06:21:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3E534299D5; Wed, 20 Mar 2019 06:21:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E12FA299D3 for ; Wed, 20 Mar 2019 06:21:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727482AbfCTGVN (ORCPT ); Wed, 20 Mar 2019 02:21:13 -0400 Received: from mail-oi1-f196.google.com ([209.85.167.196]:40454 "EHLO mail-oi1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727439AbfCTGVM (ORCPT ); Wed, 20 Mar 2019 02:21:12 -0400 Received: by mail-oi1-f196.google.com with SMTP id 3so259256oir.7 for ; Tue, 19 Mar 2019 23:21:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ca9SiysRDsF5dllhJ9lk+Pw7elzNj7M38SaL4G+4s/o=; b=VFyVdgbqRMNEeobUn1XwHQdQJTeI9Juz2EhfiXwqxpI0Llo0TFP+HpARVexaKkgAIg 0OdreSVQyZPbdrbuWNDWFphFhEbfSaxMvBGw8/tuoJcQM5oAnM0Wmea7nUfBtdYPhU21 LOsFpXNpBa4ex0Z+2NKG/3IU5qvgH03xVbuZS7PN0znunSzbZhFJcZjfjTJBtv8/S5QQ S8ErZWi8pkS0z6hRb6AyyaKphPdjlycDeWZzHAZzvsjAUScOcVdGJSLwcqLu4FfU6UAv Aa+x8k/XTs1lAQu4tAeXVdXc+rXYsarrOVvSTC9C1nJZx1MpKyjptij8dD3vCphzCLvC rcbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ca9SiysRDsF5dllhJ9lk+Pw7elzNj7M38SaL4G+4s/o=; b=KeY8gcObbYKZ30neIelkGYaPgUiWnDoVCCDF9QoMLNcBFOfkuDo/OqxxFGDvfjOcnA 7btMbzz/WfzY1+IhmasPCpF4jebgCpKRK9ghQnZ3gxETbGqEz/j7fGLb8vHFgJuxjlcf gQZLrn3puNMLPrB58N4E6pKSMjItxKvzvp3Ol6ZedPsQpuM3UcNnKSm4yuG5qq8VEtrQ QvFKavpMi+hcxAMmCBclhmvulkZXyfhbqCQqhnnSt5lIbm7c1opuvy80C/TtHPw778bG DrQkPKXSk1KVpfsS60pRSD50bJuCrj4YYq7kyqJvtwInKpI3zmqo3KMZrGw/s9RwJOp/ MV2g== X-Gm-Message-State: APjAAAXcRl4glVGT9r4IZqgfDkN/BCbbYyZc0TX6vSLkeQswWTHvuMQx tpZKgqATvxnU9Cy7BMzVI1Lr6T9QFjE= X-Google-Smtp-Source: APXvYqwa5TIeGkiCkAALg+cS3HeixMbF6MXc8EfPN7933QlOF3KzZKznBqULdoJjaW/RtvXEw0n6Mg== X-Received: by 2002:aca:f581:: with SMTP id t123mr4374147oih.0.1553062871745; Tue, 19 Mar 2019 23:21:11 -0700 (PDT) Received: from localhost.localdomain (li808-42.members.linode.com. [104.237.132.42]) by smtp.gmail.com with ESMTPSA id e186sm401517oia.44.2019.03.19.23.21.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Mar 2019 23:21:10 -0700 (PDT) From: Leo Yan To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, Will Deacon , Marc Zyngier , Jean-Philippe Brucker , Eric Auger , Robin Murphy Cc: Leo Yan Subject: [PATCH kvmtool v2 1/3] vfio-pci: Release INTx's unmask eventfd properly Date: Wed, 20 Mar 2019 14:20:44 +0800 Message-Id: <20190320062046.3895-2-leo.yan@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190320062046.3895-1-leo.yan@linaro.org> References: <20190320062046.3895-1-leo.yan@linaro.org> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The PCI device INTx uses event fd 'unmask_fd' to signal the deassertion of the line from guest to host; but this eventfd isn't released properly when disable INTx. This patch firstly adds field 'unmask_fd' in struct vfio_pci_device for storing unmask eventfd and close it when disable INTx. Signed-off-by: Leo Yan Reviewed-by: Jean-Philippe Brucker --- include/kvm/vfio.h | 1 + vfio/pci.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/include/kvm/vfio.h b/include/kvm/vfio.h index 60e6c54..28223cf 100644 --- a/include/kvm/vfio.h +++ b/include/kvm/vfio.h @@ -74,6 +74,7 @@ struct vfio_pci_device { unsigned long irq_modes; int intx_fd; + int unmask_fd; unsigned int intx_gsi; struct vfio_pci_msi_common msi; struct vfio_pci_msi_common msix; diff --git a/vfio/pci.c b/vfio/pci.c index 03de3c1..5224fee 100644 --- a/vfio/pci.c +++ b/vfio/pci.c @@ -1008,6 +1008,7 @@ static void vfio_pci_disable_intx(struct kvm *kvm, struct vfio_device *vdev) irq__del_irqfd(kvm, gsi, pdev->intx_fd); close(pdev->intx_fd); + close(pdev->unmask_fd); } static int vfio_pci_enable_intx(struct kvm *kvm, struct vfio_device *vdev) @@ -1095,6 +1096,7 @@ static int vfio_pci_enable_intx(struct kvm *kvm, struct vfio_device *vdev) } pdev->intx_fd = trigger_fd; + pdev->unmask_fd = unmask_fd; /* Guest is going to ovewrite our irq_line... */ pdev->intx_gsi = gsi; From patchwork Wed Mar 20 06:20:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leo Yan X-Patchwork-Id: 10860881 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B1B90139A for ; Wed, 20 Mar 2019 06:21:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 99586299D3 for ; Wed, 20 Mar 2019 06:21:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8CB7A299D5; Wed, 20 Mar 2019 06:21:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 38904299D3 for ; Wed, 20 Mar 2019 06:21:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727453AbfCTGVQ (ORCPT ); Wed, 20 Mar 2019 02:21:16 -0400 Received: from mail-oi1-f194.google.com ([209.85.167.194]:44126 "EHLO mail-oi1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727505AbfCTGVQ (ORCPT ); Wed, 20 Mar 2019 02:21:16 -0400 Received: by mail-oi1-f194.google.com with SMTP id i21so884940oib.11 for ; Tue, 19 Mar 2019 23:21:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=clpCQzSBoguvAsOQb5+AHrmE6H+MysxHikfRWT15AjI=; b=oyqmhm8wqdYmeW49pOJZx6kWHiKDVsu6mO0AtDB3KWaqry+VChCSRX11ylVZqzq0x3 L+bzAwk9OPL1vXQx1VzemuDFhgA2wRwCNbt5utrTqy/a5hlpRuWpSX1bvqElwUhATxcF VE9hb/z7qo4waRtOQ6YNoXbWGZagx6h/N4PTRX+elfg8P/Tee1hZowuDJBN2pOJcSxPQ 38q/+wd0aZ1nviTKGTOTqNyTJ3ZMxfTvZ6BzbU6ADMes4R39Jkj7TvhRPto3gFd97z8c OZK3Ds1wslc2QGASW2RW8hxBXJsDweuO7VGZ7TkOtUbbFgWgzgx2aeFm2G6RSlBT9AB9 mztQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=clpCQzSBoguvAsOQb5+AHrmE6H+MysxHikfRWT15AjI=; b=VDQdQr8dqJcDeUu/X5D9FPHxlVuTmEMJ7eVKg6WgasJ3ySLQlc9mM6nIcGuEz+BLPW nVy1nTPDYqY5QT7BLMchALX4oXEeUhOnAUqXwaxel95QupDozEEE2agj2lCQcl7qFBQu pmHyZgS5oCOA19283krSKPZBpVbowXu5kMKNPhQjDyMRUc0T54pDyxbGlTGX7nWPvPsB WU3Hl0g4bbA9X/TBBeXTHME1R4ychNjQJYRyBlZjWqXDxozdylwWA+PyB2h3sAKJdw+t 2ftBVMQq7wfeXP0RtytwjqdUzP/AecWS4UA85w4j3tdLfExiz4/hHb5hY6yQhSsd29Vr GTOQ== X-Gm-Message-State: APjAAAXDuBfT8SFylbZbx6Ga8w7Mto0mZWbVdmKVNKwkQl+Cbq26bSK0 tOIAxxKmHyNiESn+oTv1HnkAJMWpg5g= X-Google-Smtp-Source: APXvYqzw2h4pPAz5LzuuSDP5LAM0YGg9FPNgY4lM6SR6vVWkTXOJP/i8IYs8HXa5FDwdWFK2TE2fOA== X-Received: by 2002:aca:cf96:: with SMTP id f144mr4112569oig.19.1553062875372; Tue, 19 Mar 2019 23:21:15 -0700 (PDT) Received: from localhost.localdomain (li808-42.members.linode.com. [104.237.132.42]) by smtp.gmail.com with ESMTPSA id e186sm401517oia.44.2019.03.19.23.21.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Mar 2019 23:21:14 -0700 (PDT) From: Leo Yan To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, Will Deacon , Marc Zyngier , Jean-Philippe Brucker , Eric Auger , Robin Murphy Cc: Leo Yan Subject: [PATCH kvmtool v2 2/3] vfio-pci: Remove useless FDs reservation in vfio_pci_enable_intx() Date: Wed, 20 Mar 2019 14:20:45 +0800 Message-Id: <20190320062046.3895-3-leo.yan@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190320062046.3895-1-leo.yan@linaro.org> References: <20190320062046.3895-1-leo.yan@linaro.org> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Since INTx only uses 2 FDs, it's not particularly useful to reserve FDs in function vfio_pci_enable_intx(); so this patch is to remove FDs reservation in this function. Signed-off-by: Leo Yan --- vfio/pci.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/vfio/pci.c b/vfio/pci.c index 5224fee..d025581 100644 --- a/vfio/pci.c +++ b/vfio/pci.c @@ -1025,8 +1025,6 @@ static int vfio_pci_enable_intx(struct kvm *kvm, struct vfio_device *vdev) .index = VFIO_PCI_INTX_IRQ_INDEX, }; - vfio_pci_reserve_irq_fds(2); - ret = ioctl(vdev->fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info); if (ret || irq_info.count == 0) { vfio_dev_err(vdev, "no INTx reported by VFIO"); From patchwork Wed Mar 20 06:20:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leo Yan X-Patchwork-Id: 10860883 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 72A911575 for ; Wed, 20 Mar 2019 06:21:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 57312299D3 for ; Wed, 20 Mar 2019 06:21:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4AD0E299D5; Wed, 20 Mar 2019 06:21:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B2790299D3 for ; Wed, 20 Mar 2019 06:21:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727512AbfCTGVU (ORCPT ); Wed, 20 Mar 2019 02:21:20 -0400 Received: from mail-ot1-f67.google.com ([209.85.210.67]:34297 "EHLO mail-ot1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727518AbfCTGVU (ORCPT ); Wed, 20 Mar 2019 02:21:20 -0400 Received: by mail-ot1-f67.google.com with SMTP id r19so1138354otn.1 for ; Tue, 19 Mar 2019 23:21:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=GITb0xWl1Z9utyhYYoG1NuN7yxh2JF4G4hy7ux2h46o=; b=cE5nEg3pKkLqW+4qRtMe6VcxtSMjoNi7R8dHw7mdQKD7goioBqYxUWYlbBMkNrX8Up hS4FEXMt977Twy6hxa7L5qA/fCA50+5YUG8ileVNqaqiCF7ojGYfOFSAsRdyAPo144Ns 537+m6vnkBjK/5XHTsKN3RFvYDUGWiQsJYtb/HT1W3qsrZX7Ems7qUCXxwmS2haN/fCK c6mgh0gLIqos4KsEdcX97roQ0cZjYpW8OGx6Fy/nkbSdfK/C99GPpuK5UMB9ORLElPDf vc1zuQwdOp/oDZCcS/phV20MRZxMOXHf1qt/Sr19krROPt7TKmiOpoiIEkAAnqqH5cMo F0BQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=GITb0xWl1Z9utyhYYoG1NuN7yxh2JF4G4hy7ux2h46o=; b=jcv7Gkixs8SMFPtHVc7BnCjF8ClC5zAd8IpB23pZyWWYFRsujLCmkXafYgPYvf16rl aUbd5LCtDfZOT7hYqhXlkbmoQL/aLzyoRokd0LkcXnJhkMQ9z064n/IOpSnS3wBR2GH0 /uATWKfa/pImxQG/7MKt7L2dk5IH/BoPLKNUO2iuw4eh64YI0jqOgElkQS+skHJjynvX HhZGwBkLcG8VwivqTqtc7TDZaq3pBpGflU+ht4m+nk0lQKAvu7rzl3n4ReQNkafNCU1v blJzVNxK6F6uN5omIkZbTIQZ5cAh+E2m9Ku3huLO3tyN3za/NdoEeNhFIwV98szFINxk dFMw== X-Gm-Message-State: APjAAAXLBEWz+KmFX2gGcesfg05EljxZnHQW6ML0MYc1OdiJ05v28NPO qjmIKyfXHvR/R/IXdSqOKhDc8kna53k= X-Google-Smtp-Source: APXvYqzTlVclefO9sMW0xxsq9h56sXdO2K/uPNA07d7w3VuzYEJ/bAiVBNoBi/EQdNvxZrd5nRDNZw== X-Received: by 2002:a9d:4c9a:: with SMTP id m26mr4313837otf.369.1553062878672; Tue, 19 Mar 2019 23:21:18 -0700 (PDT) Received: from localhost.localdomain (li808-42.members.linode.com. [104.237.132.42]) by smtp.gmail.com with ESMTPSA id e186sm401517oia.44.2019.03.19.23.21.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Mar 2019 23:21:17 -0700 (PDT) From: Leo Yan To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, Will Deacon , Marc Zyngier , Jean-Philippe Brucker , Eric Auger , Robin Murphy Cc: Leo Yan Subject: [PATCH kvmtool v2 3/3] vfio-pci: Re-enable INTx mode when disable MSI/MSIX Date: Wed, 20 Mar 2019 14:20:46 +0800 Message-Id: <20190320062046.3895-4-leo.yan@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190320062046.3895-1-leo.yan@linaro.org> References: <20190320062046.3895-1-leo.yan@linaro.org> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Since PCI forbids enabling INTx, MSI or MSIX at the same time, it's by default to disable INTx mode when enable MSI/MSIX mode; but this logic is easily broken if the guest PCI driver detects the MSI/MSIX cannot work as expected and tries to rollback to use INTx mode. The INTx mode has been disabled and it has no chance to be enabled again, thus both INTx mode and MSI/MSIX mode will not be enabled in vfio for this case. Below shows the detailed flow for introducing this issue: vfio_pci_configure_dev_irqs() `-> vfio_pci_enable_intx() vfio_pci_enable_msis() `-> vfio_pci_disable_intx() vfio_pci_disable_msis() => Guest PCI driver disables MSI To fix this issue, when disable MSI/MSIX we need to check if INTx mode is available for this device or not; if the device can support INTx then we need to re-enable it so the device can fallback to use it. In this patch, should note two minor changes: - vfio_pci_disable_intx() may be called multiple times (each time the guest enables one MSI vector). This patch changes to use 'intx_fd == -1' to denote the INTx disabled, vfio_pci_disable_intx() and vfio_pci_enable_intx will directly bail out when detect INTx has been disabled and enabled respectively. - Since pci_device_header will be corrupted after PCI configuration and all irq related info will be lost. Before re-enabling INTx mode, this patch restores 'irq_pin' and 'irq_line' fields in struct pci_device_header. Signed-off-by: Leo Yan --- vfio/pci.c | 59 ++++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 48 insertions(+), 11 deletions(-) diff --git a/vfio/pci.c b/vfio/pci.c index d025581..ba971eb 100644 --- a/vfio/pci.c +++ b/vfio/pci.c @@ -28,6 +28,7 @@ struct vfio_irq_eventfd { msi_update_state(state, val, VFIO_PCI_MSI_STATE_EMPTY) static void vfio_pci_disable_intx(struct kvm *kvm, struct vfio_device *vdev); +static int vfio_pci_enable_intx(struct kvm *kvm, struct vfio_device *vdev); static int vfio_pci_enable_msis(struct kvm *kvm, struct vfio_device *vdev, bool msix) @@ -50,17 +51,14 @@ static int vfio_pci_enable_msis(struct kvm *kvm, struct vfio_device *vdev, if (!msi_is_enabled(msis->virt_state)) return 0; - if (pdev->irq_modes & VFIO_PCI_IRQ_MODE_INTX) { - /* - * PCI (and VFIO) forbids enabling INTx, MSI or MSIX at the same - * time. Since INTx has to be enabled from the start (we don't - * have a reliable way to know when the user starts using it), - * disable it now. - */ + /* + * PCI (and VFIO) forbids enabling INTx, MSI or MSIX at the same + * time. Since INTx has to be enabled from the start (after enabling + * 'pdev->intx_fd' will be assigned to an eventfd and doesn't equal + * to the init value -1), disable it now. + */ + if (pdev->irq_modes & VFIO_PCI_IRQ_MODE_INTX) vfio_pci_disable_intx(kvm, vdev); - /* Permanently disable INTx */ - pdev->irq_modes &= ~VFIO_PCI_IRQ_MODE_INTX; - } eventfds = (void *)msis->irq_set + sizeof(struct vfio_irq_set); @@ -162,7 +160,34 @@ static int vfio_pci_disable_msis(struct kvm *kvm, struct vfio_device *vdev, msi_set_enabled(msis->phys_state, false); msi_set_empty(msis->phys_state, true); - return 0; + /* + * When MSI or MSIX is disabled, this might be called when + * PCI driver detects the MSI interrupt failure and wants to + * rollback to INTx mode. Thus enable INTx if the device + * supports INTx mode in this case. + */ + if (pdev->irq_modes & VFIO_PCI_IRQ_MODE_INTX) { + /* + * Struct pci_device_header is not only used for header, + * it also is used for PCI configuration; and in the function + * vfio_pci_cfg_write() it firstly writes configuration space + * and then read back the configuration space data into the + * header structure; thus 'irq_pin' and 'irq_line' in the + * header will be overwritten. + * + * If want to enable INTx mode properly, firstly needs to + * restore 'irq_pin' and 'irq_line' values; we can simply set 1 + * to 'irq_pin', and 'pdev->intx_gsi' keeps gsi value when + * enable INTx mode previously so we can simply use it to + * recover irq line number by adding offset KVM_IRQ_OFFSET. + */ + pdev->hdr.irq_pin = 1; + pdev->hdr.irq_line = pdev->intx_gsi + KVM_IRQ_OFFSET; + + ret = vfio_pci_enable_intx(kvm, vdev); + } + + return ret >= 0 ? 0 : ret; } static int vfio_pci_update_msi_entry(struct kvm *kvm, struct vfio_device *vdev, @@ -1002,6 +1027,10 @@ static void vfio_pci_disable_intx(struct kvm *kvm, struct vfio_device *vdev) .index = VFIO_PCI_INTX_IRQ_INDEX, }; + /* INTx mode has been disabled */ + if (pdev->intx_fd == -1) + return; + pr_debug("user requested MSI, disabling INTx %d", gsi); ioctl(vdev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); @@ -1009,6 +1038,7 @@ static void vfio_pci_disable_intx(struct kvm *kvm, struct vfio_device *vdev) close(pdev->intx_fd); close(pdev->unmask_fd); + pdev->intx_fd = -1; } static int vfio_pci_enable_intx(struct kvm *kvm, struct vfio_device *vdev) @@ -1025,6 +1055,10 @@ static int vfio_pci_enable_intx(struct kvm *kvm, struct vfio_device *vdev) .index = VFIO_PCI_INTX_IRQ_INDEX, }; + /* INTx mode has been enabled */ + if (pdev->intx_fd != -1) + return 0; + ret = ioctl(vdev->fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info); if (ret || irq_info.count == 0) { vfio_dev_err(vdev, "no INTx reported by VFIO"); @@ -1140,6 +1174,9 @@ static int vfio_pci_configure_dev_irqs(struct kvm *kvm, struct vfio_device *vdev return ret; } + /* Use intx_fd=-1 to denote INTx is disabled */ + pdev->intx_fd = -1; + if (pdev->irq_modes & VFIO_PCI_IRQ_MODE_INTX) ret = vfio_pci_enable_intx(kvm, vdev);