From patchwork Fri Mar 15 08:33:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leo Yan X-Patchwork-Id: 10854265 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 90D0014DE for ; Fri, 15 Mar 2019 08:33:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7AE7D2A8B9 for ; Fri, 15 Mar 2019 08:33:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6ED982A8BB; Fri, 15 Mar 2019 08:33:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 165172A8B9 for ; Fri, 15 Mar 2019 08:33:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727172AbfCOId3 (ORCPT ); Fri, 15 Mar 2019 04:33:29 -0400 Received: from mail-oi1-f195.google.com ([209.85.167.195]:39910 "EHLO mail-oi1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726856AbfCOId3 (ORCPT ); Fri, 15 Mar 2019 04:33:29 -0400 Received: by mail-oi1-f195.google.com with SMTP id b4so6658848oif.6 for ; Fri, 15 Mar 2019 01:33:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id; bh=1zMvsKV+IZjO1EXSt7cpNuVogQUC2G1GOs9flQ0xu6s=; b=t3nL9azEi3/1u8IwBZS7nh/FmM2B6PP8YxoHuOHO+rKnPzv1txCfjR6cNX52LAWCCM mj8C4G94b8dPIBp3bYU66hyybAJPjl1fQ0cm8RorXBcrAIfUei3q9bekpNvi7W0mIvbu NfASpnKzSOGWPgTycD4Ln/HWQIjkf4DfSYxiReaj6dxm34M8mk+Mq6vMOp5zzBT+4mGb Wt7+ViTtdtCUMNirSSnssWTKiVAqZ1PmZwzofjCKwR24CnFnyY+v5LZo0P9+cgXSyD7i NbrDImB1/HKIUrXSctTqKAPNknwfJfw/7/s58WOMl6UtrQ9QwlEyEqWyrn9P3JcFw0kP pHfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=1zMvsKV+IZjO1EXSt7cpNuVogQUC2G1GOs9flQ0xu6s=; b=aaIEH7aRQHEUBOM5EMxJQ3m5Rw35TKB8mGuvkS/968t/r5weBPUuG6eK/9zSZU+mJQ 6lbAvwtY/Hd6+XgcaEHUpxUuv1tqYtfj+hNe2+sa5BD+/yd8WNZ+ai9rKdeIN0SzHYil EmxqErE4iXrQ6lXf0B369B/OYVR3thhywyBRFKPKaumwwkUpdSRKHQldxU6NBZPGtX6g O8g1WQPZM5ZlMrke4y6fvCGmM7PQG9jNkVmjBPlUUegoKJM3bmkw0ZKuo6u7MMNTOUdA G/Hw20Cogli+WJewby6GcIuIC/caGLAQJy+C43woewRb+lWFlhZ0u8EKt70D5UMkxl85 ryzQ== X-Gm-Message-State: APjAAAXmONyKiot8aB+kNra/HBzaKca6mHUy3/EhjbM6tBBgNvPSGlSK lxtLEm1LN6QFBexamTUuk1ISeAYgEec= X-Google-Smtp-Source: APXvYqw9ZIxg/7gd4EV9T30hP5xYQUsjL21p4JdMyLXEhD+pTJFjoqEx+27pkPlpMyD7EUtqMl8Scw== X-Received: by 2002:aca:4c10:: with SMTP id z16mr837865oia.53.1552638808435; Fri, 15 Mar 2019 01:33:28 -0700 (PDT) Received: from localhost.localdomain (li808-42.members.linode.com. [104.237.132.42]) by smtp.gmail.com with ESMTPSA id s126sm555279oia.24.2019.03.15.01.33.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 15 Mar 2019 01:33:27 -0700 (PDT) From: Leo Yan To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, Will Deacon , Marc Zyngier , Jean-Philippe Brucker Cc: Leo Yan Subject: [PATCH kvmtool v1 1/2] vfio-pci: Release INTx's guest to host eventfd properly Date: Fri, 15 Mar 2019 16:33:14 +0800 Message-Id: <20190315083315.19221-1-leo.yan@linaro.org> X-Mailer: git-send-email 2.17.1 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The PCI device INTx uses event fd 'unmask_fd' to signal the deassertion of the line from guest to host; but this eventfd isn't released properly when disable INTx. When disable INTx this patch firstly unbinds interrupt signal by calling ioctl VFIO_DEVICE_SET_IRQS and then it uses the new added field 'unmask_fd' in struct vfio_pci_device to close event fd. Signed-off-by: Leo Yan --- include/kvm/vfio.h | 1 + vfio/pci.c | 15 ++++++++++++--- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/include/kvm/vfio.h b/include/kvm/vfio.h index 60e6c54..28223cf 100644 --- a/include/kvm/vfio.h +++ b/include/kvm/vfio.h @@ -74,6 +74,7 @@ struct vfio_pci_device { unsigned long irq_modes; int intx_fd; + int unmask_fd; unsigned int intx_gsi; struct vfio_pci_msi_common msi; struct vfio_pci_msi_common msix; diff --git a/vfio/pci.c b/vfio/pci.c index 03de3c1..c0683f6 100644 --- a/vfio/pci.c +++ b/vfio/pci.c @@ -996,18 +996,26 @@ static void vfio_pci_disable_intx(struct kvm *kvm, struct vfio_device *vdev) { struct vfio_pci_device *pdev = &vdev->pci; int gsi = pdev->intx_gsi; - struct vfio_irq_set irq_set = { - .argsz = sizeof(irq_set), + struct vfio_irq_set trigger_irq = { + .argsz = sizeof(trigger_irq), .flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER, .index = VFIO_PCI_INTX_IRQ_INDEX, }; + struct vfio_irq_set unmask_irq = { + .argsz = sizeof(unmask_irq), + .flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_UNMASK, + .index = VFIO_PCI_INTX_IRQ_INDEX, + }; + pr_debug("user requested MSI, disabling INTx %d", gsi); - ioctl(vdev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + ioctl(vdev->fd, VFIO_DEVICE_SET_IRQS, &trigger_irq); + ioctl(vdev->fd, VFIO_DEVICE_SET_IRQS, &unmask_irq); irq__del_irqfd(kvm, gsi, pdev->intx_fd); close(pdev->intx_fd); + close(pdev->unmask_fd); } static int vfio_pci_enable_intx(struct kvm *kvm, struct vfio_device *vdev) @@ -1095,6 +1103,7 @@ static int vfio_pci_enable_intx(struct kvm *kvm, struct vfio_device *vdev) } pdev->intx_fd = trigger_fd; + pdev->unmask_fd = unmask_fd; /* Guest is going to ovewrite our irq_line... */ pdev->intx_gsi = gsi; From patchwork Fri Mar 15 08:33:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leo Yan X-Patchwork-Id: 10854267 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1E39714DE for ; Fri, 15 Mar 2019 08:33:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 08CA52A8B9 for ; Fri, 15 Mar 2019 08:33:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F04BA2A8BB; Fri, 15 Mar 2019 08:33:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 90E122A8B9 for ; Fri, 15 Mar 2019 08:33:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727287AbfCOIdd (ORCPT ); Fri, 15 Mar 2019 04:33:33 -0400 Received: from mail-ot1-f65.google.com ([209.85.210.65]:46451 "EHLO mail-ot1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727183AbfCOIdc (ORCPT ); Fri, 15 Mar 2019 04:33:32 -0400 Received: by mail-ot1-f65.google.com with SMTP id c18so7626539otl.13 for ; Fri, 15 Mar 2019 01:33:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=vKu92hjP8eFmmyYYR13WjPELdfrY4kxxooGFZQxsSUk=; b=PaXQg3NzHeNgbxHHa9S2FocC+IamS1eL3iRBKzlcSuVIZ6fcUQizHPEAzyZt/QrkcH H1q8KKC9ky0Fn1gVrAuecLImW47Zgg/rugeW+UdN+uES9TjsftW+NNntcB7Z8fLu2y5l rdAolI39JMsMI7A6kwPIhcd2SssnQOC5Gp6Olt2dCVK35pB9NTscKjuA8wG5G1N3E3HR LtXnxo5wSVVjwoXUmhTUWbpLtvFtdWtPyMPwrkEiDqRjlk9VJuOpMUwNgnRUaGRk1ihi HFHNVPVHYX5iG2EE/7TfR8TbDk6sLjX7wTLt6hsxQcY0dLcbJ9NSIBo9OGhF2upOi9x0 JQbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=vKu92hjP8eFmmyYYR13WjPELdfrY4kxxooGFZQxsSUk=; b=FB6HTSJo2P04Yb68UKfB/p+IZMHB5QbajFrbV2KBWexXcgU4CMPiVYFe+l+slcZzv6 fAQrbWuIkH7wnIRitDlG1fHr9Zb2YCsBYasM2gUvCn78mtGrfBASq5DLUjf5thT9pWNV 7vpap9LduP6qtW0Dytdst8898hwmsPS+3JLMk0mpegxWKArfaf3cvw77NVPCgJIRDrVh vRibK0/Fi/os0eVjXgoB8qJcvZvpGk1GBhV7KjzQUPks6fIbCRfZiu9Vw65Ik7+Ac6ek 3uWusovDHuxBTycUDbZ1c9nJ8o34YH7+3tRa9sFCV9DutGzZwzjtg0t8oTvszBJZfgHV gr+Q== X-Gm-Message-State: APjAAAXIHtrwpwbf8k5MAUIYQprIZLrVMX1kzD7yvwQZe9x/mLVDj2yV nff0yxFwUnYgyBfHKz5POw2RvifxmNY= X-Google-Smtp-Source: APXvYqy0wg1cOO+4foc1Frvjy4RZbqx+3XIVT92CM1OO43psmXbfgZZcnBYnX6AeNCY++t7qaI9vLg== X-Received: by 2002:a9d:2c44:: with SMTP id f62mr1463369otb.219.1552638811228; Fri, 15 Mar 2019 01:33:31 -0700 (PDT) Received: from localhost.localdomain (li808-42.members.linode.com. [104.237.132.42]) by smtp.gmail.com with ESMTPSA id s126sm555279oia.24.2019.03.15.01.33.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 15 Mar 2019 01:33:30 -0700 (PDT) From: Leo Yan To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, Will Deacon , Marc Zyngier , Jean-Philippe Brucker Cc: Leo Yan Subject: [PATCH kvmtool v1 2/2] vfio-pci: Fallback to INTx mode when disable MSI/MSIX Date: Fri, 15 Mar 2019 16:33:15 +0800 Message-Id: <20190315083315.19221-2-leo.yan@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190315083315.19221-1-leo.yan@linaro.org> References: <20190315083315.19221-1-leo.yan@linaro.org> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Since PCI forbids enabling INTx, MSI or MSIX at the same time, it's by default to disable INTx mode when enable MSI/MSIX mode; but this logic is easily broken if the guest PCI driver detects the MSI/MSIX cannot work as expected and tries to rollback to use INTx mode. The INTx mode has been disabled and it has no chance to be enabled again, thus both INTx mode and MSI/MSIX mode will not be enabled in vfio for this case. Below shows the detailed flow for introducing this issue: vfio_pci_configure_dev_irqs() `-> vfio_pci_enable_intx() vfio_pci_enable_msis() `-> vfio_pci_disable_intx() vfio_pci_disable_msis() => Guest PCI driver disables MSI To fix this issue, when disable MSI/MSIX we need to check if INTx mode is available for this device or not; if the device can support INTx then we need to re-enable it so the device can fallback to use it. Signed-off-by: Leo Yan --- vfio/pci.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/vfio/pci.c b/vfio/pci.c index c0683f6..44727bb 100644 --- a/vfio/pci.c +++ b/vfio/pci.c @@ -28,6 +28,7 @@ struct vfio_irq_eventfd { msi_update_state(state, val, VFIO_PCI_MSI_STATE_EMPTY) static void vfio_pci_disable_intx(struct kvm *kvm, struct vfio_device *vdev); +static int vfio_pci_enable_intx(struct kvm *kvm, struct vfio_device *vdev); static int vfio_pci_enable_msis(struct kvm *kvm, struct vfio_device *vdev, bool msix) @@ -50,7 +51,7 @@ static int vfio_pci_enable_msis(struct kvm *kvm, struct vfio_device *vdev, if (!msi_is_enabled(msis->virt_state)) return 0; - if (pdev->irq_modes & VFIO_PCI_IRQ_MODE_INTX) { + if (pdev->irq_modes & VFIO_PCI_IRQ_MODE_INTX) /* * PCI (and VFIO) forbids enabling INTx, MSI or MSIX at the same * time. Since INTx has to be enabled from the start (we don't @@ -58,9 +59,6 @@ static int vfio_pci_enable_msis(struct kvm *kvm, struct vfio_device *vdev, * disable it now. */ vfio_pci_disable_intx(kvm, vdev); - /* Permanently disable INTx */ - pdev->irq_modes &= ~VFIO_PCI_IRQ_MODE_INTX; - } eventfds = (void *)msis->irq_set + sizeof(struct vfio_irq_set); @@ -162,7 +160,16 @@ static int vfio_pci_disable_msis(struct kvm *kvm, struct vfio_device *vdev, msi_set_enabled(msis->phys_state, false); msi_set_empty(msis->phys_state, true); - return 0; + if (pdev->irq_modes & VFIO_PCI_IRQ_MODE_INTX) + /* + * When MSI or MSIX is disabled, this might be called when + * PCI driver detects the MSI interrupt failure and wants to + * rollback to INTx mode. Thus enable INTx if the device + * supports INTx mode in this case. + */ + ret = vfio_pci_enable_intx(kvm, vdev); + + return ret >= 0 ? 0 : ret; } static int vfio_pci_update_msi_entry(struct kvm *kvm, struct vfio_device *vdev,