From patchwork Fri Aug 18 04:10:20 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 9907577 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id CFF0560382 for ; Fri, 18 Aug 2017 04:10:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BA2E328B6A for ; Fri, 18 Aug 2017 04:10:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AECBA28B62; Fri, 18 Aug 2017 04:10:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4485D28BBA for ; Fri, 18 Aug 2017 04:10:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1750773AbdHREKX (ORCPT ); Fri, 18 Aug 2017 00:10:23 -0400 Received: from mx1.redhat.com ([209.132.183.28]:51368 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750705AbdHREKW (ORCPT ); Fri, 18 Aug 2017 00:10:22 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 67E88C04B927; Fri, 18 Aug 2017 04:10:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 67E88C04B927 Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=alex.williamson@redhat.com Received: from gimli.home (ovpn-116-27.phx2.redhat.com [10.3.116.27]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2D5284AA; Fri, 18 Aug 2017 04:10:20 +0000 (UTC) Subject: [PATCH] vfio: Stall vfio_del_group_dev() for container group detach From: Alex Williamson To: alex.williamson@redhat.com Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Date: Thu, 17 Aug 2017 22:10:20 -0600 Message-ID: <20170818040952.13026.24635.stgit@gimli.home> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Fri, 18 Aug 2017 04:10:22 +0000 (UTC) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When the user unbinds the last device of a group from a vfio bus driver, the devices within that group should be available for other purposes. We currently have a race that makes this generally, but not always true. The device can be unbound from the vfio bus driver, but remaining IOMMU context of the group attached to the container can result in errors as the next driver configures DMA for the device. Wait for the group to be detached from the IOMMU backend before allowing the bus driver remove callback to complete. Signed-off-by: Alex Williamson --- drivers/vfio/vfio.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c index 330d50582f40..fb78e0becd1b 100644 --- a/drivers/vfio/vfio.c +++ b/drivers/vfio/vfio.c @@ -85,6 +85,7 @@ struct vfio_group { struct list_head unbound_list; struct mutex unbound_lock; atomic_t opened; + wait_queue_head_t container_q; bool noiommu; struct kvm *kvm; struct blocking_notifier_head notifier; @@ -337,6 +338,7 @@ static struct vfio_group *vfio_create_group(struct iommu_group *iommu_group) mutex_init(&group->unbound_lock); atomic_set(&group->container_users, 0); atomic_set(&group->opened, 0); + init_waitqueue_head(&group->container_q); group->iommu_group = iommu_group; #ifdef CONFIG_VFIO_NOIOMMU group->noiommu = (iommu_group_get_iommudata(iommu_group) == &noiommu); @@ -993,6 +995,23 @@ void *vfio_del_group_dev(struct device *dev) } } while (ret <= 0); + /* + * In order to support multiple devices per group, devices can be + * plucked from the group while other devices in the group are still + * in use. The container persists with this group and those remaining + * devices still attached. If the user creates an isolation violation + * by binding this device to another driver while the group is still in + * use, that's their fault. However, in the case of removing the last, + * or potentially the only, device in the group there can be no other + * in-use devices in the group. The user has done their due diligence + * and we should lay no claims to those devices. In order to do that, + * we need to make sure the group is detached from the container. + * Without this stall, we're potentially racing with a user process + * that may attempt to immediately bind this device to another driver. + */ + if (list_empty(&group->device_list)) + wait_event(group->container_q, !group->container); + vfio_group_put(group); return device_data; @@ -1298,6 +1317,7 @@ static void __vfio_group_unset_container(struct vfio_group *group) group->iommu_group); group->container = NULL; + wake_up(&group->container_q); list_del(&group->container_next); /* Detaching the last group deprivileges a container, remove iommu */