From patchwork Fri Mar 2 23:44:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10255863 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D1DF160211 for ; Fri, 2 Mar 2018 23:44:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C0384285E4 for ; Fri, 2 Mar 2018 23:44:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B3769285F2; Fri, 2 Mar 2018 23:44:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 181E2285E4 for ; Fri, 2 Mar 2018 23:44:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934976AbeCBXo5 (ORCPT ); Fri, 2 Mar 2018 18:44:57 -0500 Received: from mail-it0-f68.google.com ([209.85.214.68]:36023 "EHLO mail-it0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S934783AbeCBXoz (ORCPT ); Fri, 2 Mar 2018 18:44:55 -0500 Received: by mail-it0-f68.google.com with SMTP id u5so3813920itc.1; Fri, 02 Mar 2018 15:44:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=BObIRCy5lm6CKZluie2A7ajCkk0oX8gYLtCZNQMl9K8=; b=B0EqgZfgJXlLPICwTgbNgj23NwjdVfb+HHAyAds2IBV4ouPcBsgRHQGBNVgPRy0cBt LewDRQ6/Qsq8oms/lG/TJj7wubv1J8VyRnNjI2lR1N8VNkBTk9wY4DW4NKEnyp92VZLH S5ksrOYLWBSSSIqPL0/5sD7i1CMkrRQFMnnhCwThUmg6++PxKyHup891poXv4hUNQYYF /4F4la4N9zmkwzMW4tMpfzeiP7HbXLx+SAEzIDTRg7A0O17bpVGrjZTrwN2ry2/Tj/K5 QsUW4bSdgrui1FsLrfMcaniHMYFvr64QFicBPKoaoP5YA8u0XNTBwbQMh2sClkh+21+x XKow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=BObIRCy5lm6CKZluie2A7ajCkk0oX8gYLtCZNQMl9K8=; b=bTmfBr8DvuyCQfOpifL8k8GdL7GpqDwoNi4SBdykPyz+lvDBmMZAqnf+B+0IUE2R5w F2I0QXCG7WketcbryACJdn5WCBcngmRZiVVsYA52FaqQ6myMbSh55SrnmlNIK5qxWxT7 NQVTuj0N1u35FD64ZCz073GcSTJJAUilfgYKWGerLZsJ7eWGkIP++zIvvg0Ks9EP52Em FstTPL/Tz8nQLZeP1Sl/dhTTftKJWiunNNIAEYJaC0pElKAZlbJsYPFlh9nPLD0thuBu 56iBkt8dCtCJDtvh1nFlWtI8QYdkV8fkw6XdAJrc6BrN6Ik4CusDm0AfUa5yF4No8Nkd 701A== X-Gm-Message-State: AElRT7HxhOsDHskXKMq9NFXj3Rj1cZ7H+h2byJQH1ZGzUARlsFRvFKxn nelBv81XH65A58rdk5Qh81w= X-Google-Smtp-Source: AG47ELtZWqoJcU8Z0llbCajCKy+NHROpC+wtovEROqwhRGzMD+EPQ5VjE0q3OKro1DOgYunfBoS1Dg== X-Received: by 10.36.43.80 with SMTP id h77mr4725059ita.103.1520034294578; Fri, 02 Mar 2018 15:44:54 -0800 (PST) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id v8sm4700390iog.67.2018.03.02.15.44.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 02 Mar 2018 15:44:54 -0800 (PST) Subject: [PATCH 2/3] vfio: Add support for unmanaged or userspace managed SR-IOV From: Alexander Duyck To: bhelgaas@google.com, alexander.h.duyck@intel.com, linux-pci@vger.kernel.org Cc: virtio-dev@lists.oasis-open.org, kvm@vger.kernel.org, netdev@vger.kernel.org, dan.daly@intel.com, linux-kernel@vger.kernel.org, mheyne@amazon.de, liang-min.wang@intel.com, mark.d.rustad@intel.com, dwmw2@infradead.org, dwmw@amazon.co.uk Date: Fri, 02 Mar 2018 15:44:52 -0800 Message-ID: <20180302234432.3337.96123.stgit@localhost.localdomain> In-Reply-To: <20180302234218.3337.16486.stgit@localhost.localdomain> References: <20180302234218.3337.16486.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Alexander Duyck This patch is meant to allow assignment of an SR-IOV enabled PF, as in VFs have been generated, with vfio-pci. My understanding is the primary use case for this is something like DPDK running the PF while the VFs are all assigned to guests. A secondary effect of this is that it provides an interface through which it would be possible to enable SR-IOV on drivers that may not have a physical function that actually manages the device. Enabling SR-IOV should be pretty straight forward. As long as there are no userspace processes currently controlling the interface the number of VFs can be changed, and VFs will be generated without drivers being loaded on the host. Once the userspace process begins controlling the interface the number of VFs cannot be updated via the sysfs until the control is released. Note the VFs will have drivers load on them in the host if the sriov_unmanaged_autoprobe is updated to a value of 1. However the behavior of the VFs in such a setup cannot be guaranteed as the PF will not be available until the userspace process starts and begins to manage the device. For now I am leaving the value as locked when the PF is being controlled from userspace as a form of synchronization. Basically this way we cannot have the number of VFs change out from under the process so it should not require any notification framework, and the configuration can just be read out via configuration space accesses. Signed-off-by: Alexander Duyck --- drivers/vfio/pci/vfio_pci.c | 57 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c index b0f759476900..3023bda39aa9 100644 --- a/drivers/vfio/pci/vfio_pci.c +++ b/drivers/vfio/pci/vfio_pci.c @@ -1224,6 +1224,8 @@ static void vfio_pci_remove(struct pci_dev *pdev) VGA_RSRC_LEGACY_IO | VGA_RSRC_LEGACY_MEM); } + pci_disable_sriov(pdev); + if (!disable_idle_d3) pci_set_power_state(pdev, PCI_D0); } @@ -1260,12 +1262,67 @@ static pci_ers_result_t vfio_pci_aer_err_detected(struct pci_dev *pdev, .error_detected = vfio_pci_aer_err_detected, }; +static int vfio_pci_sriov_configure(struct pci_dev *pdev, int nr_virtfn) +{ + struct vfio_pci_device *vdev; + struct vfio_device *device; + int err; + + device = vfio_device_get_from_dev(&pdev->dev); + if (device == NULL) + return -ENODEV; + + vdev = vfio_device_data(device); + if (vdev == NULL) { + vfio_device_put(device); + return -ENODEV; + } + + /* + * If a userspace process is already using this device just return + * busy and don't allow for any changes. + */ + if (vdev->refcnt) { + pci_warn(pdev, + "PF is currently in use, blocked until released by user\n"); + return -EBUSY; + } + + err = pci_sriov_configure_unmanaged(pdev, nr_virtfn); + if (err <= 0) + return err; + + /* + * We are now leaving VFs in the control of some unknown PF entity. + * + * Best case is a well behaved userspace PF is expected and any VMs + * that the VFs will be assigned to are dependent on the userspace + * entity anyway. An example being NFV where maybe the PF is acting + * as an accelerated interface for a firewall or switch. + * + * Worst case is somebody really messed up and just enabled SR-IOV + * on a device they were planning to assign to a VM somwhere. + * + * In either case it is probably best for us to set the taint flag + * and warn the user since this could get really ugly really quick + * if this wasn't what they were planning to do. + */ + add_taint(TAINT_USER, LOCKDEP_STILL_OK); + pci_warn(pdev, + "Adding kernel taint for vfio-pci now managing SR-IOV PF device\n"); + + return nr_virtfn; +} + static struct pci_driver vfio_pci_driver = { .name = "vfio-pci", .id_table = NULL, /* only dynamic ids */ .probe = vfio_pci_probe, .remove = vfio_pci_remove, .err_handler = &vfio_err_handlers, +#ifdef CONFIG_PCI_IOV + .sriov_configure = vfio_pci_sriov_configure, +#endif }; struct vfio_devices {