From patchwork Tue Apr 5 11:41:57 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cao jin X-Patchwork-Id: 8749941 Return-Path: X-Original-To: patchwork-qemu-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 5135AC0553 for ; Tue, 5 Apr 2016 11:42:35 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id E6759202EC for ; Tue, 5 Apr 2016 11:42:29 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AC50C202D1 for ; Tue, 5 Apr 2016 11:42:28 +0000 (UTC) Received: from localhost ([::1]:36388 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1anPNM-00031D-2X for patchwork-qemu-devel@patchwork.kernel.org; Tue, 05 Apr 2016 07:42:28 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38860) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1anPKv-000718-8l for qemu-devel@nongnu.org; Tue, 05 Apr 2016 07:39:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1anPKt-0006HG-5w for qemu-devel@nongnu.org; Tue, 05 Apr 2016 07:39:57 -0400 Received: from [59.151.112.132] (port=2407 helo=heian.cn.fujitsu.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1anPKr-0006Et-2p for qemu-devel@nongnu.org; Tue, 05 Apr 2016 07:39:55 -0400 X-IronPort-AV: E=Sophos;i="5.22,518,1449504000"; d="scan'208";a="5281996" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 05 Apr 2016 19:39:48 +0800 Received: from G08CNEXCHPEKD03.g08.fujitsu.local (unknown [10.167.33.85]) by cn.fujitsu.com (Postfix) with ESMTP id F31AC408D279; Tue, 5 Apr 2016 19:39:44 +0800 (CST) Received: from G08FNSTD140223.g08.fujitsu.local (10.167.226.69) by G08CNEXCHPEKD03.g08.fujitsu.local (10.167.33.89) with Microsoft SMTP Server (TLS) id 14.3.279.2; Tue, 5 Apr 2016 19:39:44 +0800 From: Cao jin To: Date: Tue, 5 Apr 2016 19:41:57 +0800 Message-ID: <1459856523-17085-7-git-send-email-caoj.fnst@cn.fujitsu.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1459856523-17085-1-git-send-email-caoj.fnst@cn.fujitsu.com> References: <1459856523-17085-1-git-send-email-caoj.fnst@cn.fujitsu.com> MIME-Version: 1.0 X-Originating-IP: [10.167.226.69] X-yoursite-MailScanner-ID: F31AC408D279.AFD12 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: caoj.fnst@cn.fujitsu.com X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 59.151.112.132 Cc: Chen Fan , izumi.taku@jp.fujitsu.com, alex.williamson@redhat.com, mst@redhat.com Subject: [Qemu-devel] [patch v6 06/12] vfio: add check host bus reset is support or not X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Chen Fan When assigning a vfio device with AER enabled, we must check whether the device supports a host bus reset (ie. hot reset) as this may be used by the guest OS in order to recover the device from an AER error. QEMU must therefore have the ability to perform a physical host bus reset using the existing vfio APIs in response to a virtual bus reset in the VM. A physical bus reset affects all of the devices on the host bus, therefore we place a few simplifying configuration restriction on the VM: - All physical devices affected by a bus reset must be assigned to the VM with AER enabled on each and be configured on the same virtual bus in the VM. - No devices unaffected by the bus reset, be they physical, emulated, or paravirtual may be configured on the same virtual bus as a device supporting AER signaling through vfio. In other words users wishing to enable AER on a multifunction device need to assign all functions of the device to the same virtual bus and enable AER support for each device. The easiest way to accomplish this is to identity map the physical functions to virtual functions with multifunction enabled on the virtual device. Signed-off-by: Chen Fan --- hw/vfio/pci.c | 280 +++++++++++++++++++++++++++++++++++++++++++++++++++++----- hw/vfio/pci.h | 1 + 2 files changed, 258 insertions(+), 23 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 5b23a86..d94e643 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -1716,6 +1716,41 @@ static void vfio_check_af_flr(VFIOPCIDevice *vdev, uint8_t pos) } } +static int vfio_pci_name_to_addr(const char *name, PCIHostDeviceAddress *addr) +{ + if (strlen(name) != 12 || + sscanf(name, "%04x:%02x:%02x.%1x", &addr->domain, + &addr->bus, &addr->slot, &addr->function) != 4) { + return -EINVAL; + } + + return 0; +} + +static bool vfio_pci_host_match(PCIHostDeviceAddress *addr, const char *name) +{ + PCIHostDeviceAddress tmp; + + if (vfio_pci_name_to_addr(name, &tmp)) { + return false; + } + + return (tmp.domain == addr->domain && tmp.bus == addr->bus && + tmp.slot == addr->slot && tmp.function == addr->function); +} + +static bool vfio_pci_host_match_slot(PCIHostDeviceAddress *addr, const char *name) +{ + PCIHostDeviceAddress tmp; + + if (vfio_pci_name_to_addr(name, &tmp)) { + return false; + } + + return (tmp.domain == addr->domain && tmp.bus == addr->bus && + tmp.slot == addr->slot); +} + /* * return negative with errno, return 0 on success. * if success, the point of ret_info fill with the affected device reset info. @@ -1877,6 +1912,203 @@ static int vfio_add_std_cap(VFIOPCIDevice *vdev, uint8_t pos) return 0; } +/* + * Calculate the max function number on specified bus. + * if the bridge is not pcie bridge or support ARI, return + * 255, otherwise return 8. + */ +static int vfio_pci_bus_devfn_limit(PCIBus *bus) +{ + PCIDevice *br; + + br = pci_bridge_get_device(bus); + if (!br || + !pci_bus_is_express(bus) || + pcie_cap_is_arifwd_enabled(br)) { + return 255; + } + + return 8; +} + +static void vfio_check_hot_bus_reset(VFIOPCIDevice *vdev, Error **errp) +{ + PCIBus *bus = vdev->pdev.bus; + struct vfio_pci_hot_reset_info *info = NULL; + struct vfio_pci_dependent_device *devices; + VFIOGroup *group; + int ret, i, devfn, devfn_limit; + + ret = vfio_get_hot_reset_info(vdev, &info); + if (ret) { + error_setg(errp, "vfio: Cannot enable AER for device %s," + " device does not support hot reset.", + vdev->vbasedev.name); + return; + } + + /* List all affected devices by bus reset */ + devices = &info->devices[0]; + + /* Verify that we have all the groups required */ + for (i = 0; i < info->count; i++) { + PCIHostDeviceAddress host; + VFIOPCIDevice *tmp; + VFIODevice *vbasedev_iter; + bool found = false; + + host.domain = devices[i].segment; + host.bus = devices[i].bus; + host.slot = PCI_SLOT(devices[i].devfn); + host.function = PCI_FUNC(devices[i].devfn); + + /* Skip the current device */ + if (vfio_pci_host_match(&host, vdev->vbasedev.name)) { + continue; + } + + /* Ensure we own the group of the affected device */ + QLIST_FOREACH(group, &vfio_group_list, next) { + if (group->groupid == devices[i].group_id) { + break; + } + } + + if (!group) { + error_setg(errp, "vfio: Cannot enable AER for device %s, " + "depends on group %d which is not owned.", + vdev->vbasedev.name, devices[i].group_id); + goto out; + } + + /* Ensure affected devices for reset on the same bus */ + QLIST_FOREACH(vbasedev_iter, &group->device_list, next) { + if (vbasedev_iter->type != VFIO_DEVICE_TYPE_PCI) { + continue; + } + tmp = container_of(vbasedev_iter, VFIOPCIDevice, vbasedev); + if (vfio_pci_host_match(&host, tmp->vbasedev.name)) { + /* + * AER errors may be broadcast to all functions of a multi- + * function endpoint. If any of those sibling functions are + * also assigned, they need to have AER enabled or else an + * error may continue to cause a vm_stop condition. IOW, + * AER setup of this function would be pointless. + */ + if (vfio_pci_host_match_slot(&host, vdev->vbasedev.name) && + !(tmp->features & VFIO_FEATURE_ENABLE_AER)) { + error_setg(errp, "vfio: Cannot enable AER for device %s, on same slot" + " the dependent device %s which does not enable AER.", + vdev->vbasedev.name, tmp->vbasedev.name); + goto out; + } + + if (tmp->pdev.bus != bus) { + error_setg(errp, "vfio: Cannot enable AER for device %s, " + "the dependent device %s is not on the same bus", + vdev->vbasedev.name, tmp->vbasedev.name); + goto out; + } + found = true; + break; + } + } + + /* Ensure all affected devices assigned to VM */ + if (!found) { + error_setg(errp, "vfio: Cannot enable AER for device %s, " + "the dependent device %04x:%02x:%02x.%x " + "is not assigned to VM.", + vdev->vbasedev.name, host.domain, host.bus, + host.slot, host.function); + goto out; + } + } + + /* + * The above code verified that all devices affected by a bus reset + * exist on the same bus in the VM. To further simplify, we also + * require that there are no additional devices beyond those existing on + * the VM bus. + */ + devfn_limit = vfio_pci_bus_devfn_limit(bus); + for (devfn = 0; devfn < devfn_limit; devfn++) { + VFIOPCIDevice *tmp; + PCIDevice *dev; + bool found = false; + + dev = pci_find_device(bus, pci_bus_num(bus), devfn); + + if (!dev) { + continue; + } + + if (!object_dynamic_cast(OBJECT(dev), "vfio-pci")) { + error_setg(errp, "vfio: Cannot enable AER for device %s, device" + " %s: VM address %02x.%d cannot be configured" + " on the same virtual bus", + vdev->vbasedev.name, dev->name, + PCI_SLOT(dev->devfn), PCI_FUNC(dev->devfn)); + goto out; + } + + tmp = DO_UPCAST(VFIOPCIDevice, pdev, dev); + for (i = 0; i < info->count; i++) { + PCIHostDeviceAddress host; + + host.domain = devices[i].segment; + host.bus = devices[i].bus; + host.slot = PCI_SLOT(devices[i].devfn); + host.function = PCI_FUNC(devices[i].devfn); + + if (vfio_pci_host_match(&host, tmp->vbasedev.name)) { + found = true; + break; + } + } + + if (!found) { + error_setg(errp, "vfio: Cannot enable AER for device %s, vfio-pci" + " device %s at VM address %02x.%d cannot be" + " configured on the same virtual bus", + vdev->vbasedev.name, tmp->vbasedev.name, + PCI_SLOT(tmp->pdev.devfn), PCI_FUNC(tmp->pdev.devfn)); + goto out; + } + } + +out: + g_free(info); + return; +} + +static void vfio_aer_check_host_bus_reset(Error **errp) +{ + VFIOGroup *group; + VFIODevice *vbasedev; + VFIOPCIDevice *vdev; + Error *local_err = NULL; + + /* Check All vfio-pci devices if have bus reset capability */ + QLIST_FOREACH(group, &vfio_group_list, next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + if (vbasedev->type != VFIO_DEVICE_TYPE_PCI) { + continue; + } + vdev = container_of(vbasedev, VFIOPCIDevice, vbasedev); + if (vdev->features & VFIO_FEATURE_ENABLE_AER) { + vfio_check_hot_bus_reset(vdev, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + } + } + } + + return; +} + static int vfio_setup_aer(VFIOPCIDevice *vdev, uint8_t cap_ver, int pos, uint16_t size) { @@ -2060,29 +2292,6 @@ static void vfio_pci_post_reset(VFIOPCIDevice *vdev) vfio_intx_enable(vdev); } -static int vfio_pci_name_to_addr(const char *name, PCIHostDeviceAddress *addr) -{ - if (strlen(name) != 12 || - sscanf(name, "%04x:%02x:%02x.%1x", &addr->domain, - &addr->bus, &addr->slot, &addr->function) != 4) { - return -EINVAL; - } - - return 0; -} - -static bool vfio_pci_host_match(PCIHostDeviceAddress *addr, const char *name) -{ - PCIHostDeviceAddress tmp; - - if (vfio_pci_name_to_addr(name, &tmp)) { - return false; - } - - return (tmp.domain == addr->domain && tmp.bus == addr->bus && - tmp.slot == addr->slot && tmp.function == addr->function); -} - static int vfio_pci_hot_reset(VFIOPCIDevice *vdev, bool single) { VFIOGroup *group; @@ -2589,6 +2798,22 @@ static void vfio_unregister_req_notifier(VFIOPCIDevice *vdev) vdev->req_enabled = false; } +static void vfio_pci_machine_done_notify(Notifier *notifier, void *unused) +{ + Error *local_err = NULL; + + vfio_aer_check_host_bus_reset(&local_err); + if (local_err) { + fprintf(stderr, "%s\n", error_get_pretty(local_err)); + error_free(local_err); + exit(1); + } +} + +static Notifier machine_notifier = { + .notify = vfio_pci_machine_done_notify, +}; + static int vfio_initfn(PCIDevice *pdev) { VFIOPCIDevice *vdev = DO_UPCAST(VFIOPCIDevice, pdev, pdev); @@ -2934,6 +3159,15 @@ static const TypeInfo vfio_pci_dev_info = { static void register_vfio_pci_dev_type(void) { type_register_static(&vfio_pci_dev_info); + + /* + * The AER configuration may depend on multiple devices, so we cannot + * validate consistency after each device is initialized. We can only + * depend on function initialization order (function 0 last) for hotplug + * devices, therefore a machine-init-done notifier is used to validate + * the configuration after all cold-plug devices are processed. + */ + qemu_add_machine_init_done_notifier(&machine_notifier); } type_init(register_vfio_pci_dev_type) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 7b3924e..db7c6d5 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -15,6 +15,7 @@ #include "qemu-common.h" #include "exec/memory.h" #include "hw/pci/pci.h" +#include "hw/pci/pci_bus.h" #include "hw/pci/pci_bridge.h" #include "hw/vfio/vfio-common.h" #include "qemu/event_notifier.h"