From patchwork Fri May 19 21:57:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 13248901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 15AA4C77B75 for ; Fri, 19 May 2023 21:58:24 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q086k-0005wU-Nm; Fri, 19 May 2023 17:57:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q086j-0005wE-Fc for qemu-devel@nongnu.org; Fri, 19 May 2023 17:57:53 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q086h-0004F4-Op for qemu-devel@nongnu.org; Fri, 19 May 2023 17:57:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684533470; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RYrP0o+CAqP3kVFMwvnEaTCaiiKfYNj8nTDnmx99pDo=; b=OMk6z/hlMzlrb/jKzCUH1YVS2lcP8o9XCu8+WVE9ZzdZiLpX2poybl2OXzNj7en564J3wC ERXMAq02Zv5mge6r0aWbIe9R5hYbaLhXy7yHZGbZILhKTi1/i+38cq3MoMfXEnCgPdUUFN vPAmqKIFwCZ5BOn1awaPiX+SqsAA9g8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-467-ZA92AA6lN1uOzK4upMqo0w-1; Fri, 19 May 2023 17:57:49 -0400 X-MC-Unique: ZA92AA6lN1uOzK4upMqo0w-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AAEB4101A53B; Fri, 19 May 2023 21:57:48 +0000 (UTC) Received: from omen.home.shazbot.org (unknown [10.22.10.168]) by smtp.corp.redhat.com (Postfix) with ESMTP id 43AA8C0004B; Fri, 19 May 2023 21:57:48 +0000 (UTC) From: Alex Williamson To: robin@streamhpc.com, mst@redhat.com, marcel.apfelbaum@gmail.com Cc: Alex Williamson , qemu-devel@nongnu.org Subject: [RFC PATCH 1/4] linux-headers: Update for vfio capability reporting AtomicOps Date: Fri, 19 May 2023 15:57:36 -0600 Message-Id: <20230519215739.402729-2-alex.williamson@redhat.com> In-Reply-To: <20230519215739.402729-1-alex.williamson@redhat.com> References: <20230519215739.402729-1-alex.williamson@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass client-ip=170.10.129.124; envelope-from=alex.williamson@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This is a partial linux-headers update for illustrative and testing purposes only, NOT FOR COMMIT. Signed-off-by: Alex Williamson --- linux-headers/linux/vfio.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/linux-headers/linux/vfio.h b/linux-headers/linux/vfio.h index 4a534edbdcba..443a8851e156 100644 --- a/linux-headers/linux/vfio.h +++ b/linux-headers/linux/vfio.h @@ -240,6 +240,20 @@ struct vfio_device_info { #define VFIO_DEVICE_INFO_CAP_ZPCI_UTIL 3 #define VFIO_DEVICE_INFO_CAP_ZPCI_PFIP 4 +/* + * The following VFIO_DEVICE_INFO capability reports support for PCIe AtomicOp + * completion to the root bus with supported widths provided via flags. + */ +#define VFIO_DEVICE_INFO_CAP_PCI_ATOMIC_COMP 5 +struct vfio_device_info_cap_pci_atomic_comp { + struct vfio_info_cap_header header; + __u32 flags; +#define VFIO_PCI_ATOMIC_COMP32 (1 << 0) +#define VFIO_PCI_ATOMIC_COMP64 (1 << 1) +#define VFIO_PCI_ATOMIC_COMP128 (1 << 2) + __u32 reserved; +}; + /** * VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8, * struct vfio_region_info) From patchwork Fri May 19 21:57:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 13248904 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 77593C77B75 for ; Fri, 19 May 2023 21:58:59 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q086n-0005yu-II; Fri, 19 May 2023 17:57:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q086l-0005wf-59 for qemu-devel@nongnu.org; Fri, 19 May 2023 17:57:55 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q086j-0004FD-Gr for qemu-devel@nongnu.org; Fri, 19 May 2023 17:57:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684533472; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wZvw6QjxR2MUxzxGaroxNq9sovUgvJA+2KZO2vFMJUE=; b=ZRyOYFNhZDtdJMu6Yqs1WTVWn6POasV4mL2rB14x2SDEVoluNwm9dow8pdqs1r4Jb4+pU8 lichmv2Ye1tiTSkYC6CAYg6O7M/hpopR0BDr5jUNUvqhibMrzWb2Iz272PCMnp8wBdQz8i cfFLBWcADbukFWZy1A5ogA1aGkYyQec= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-507-2kliOwucOyWLTq6lTVPw1g-1; Fri, 19 May 2023 17:57:49 -0400 X-MC-Unique: 2kliOwucOyWLTq6lTVPw1g-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 39680101A53A; Fri, 19 May 2023 21:57:49 +0000 (UTC) Received: from omen.home.shazbot.org (unknown [10.22.10.168]) by smtp.corp.redhat.com (Postfix) with ESMTP id C3EE5C0004B; Fri, 19 May 2023 21:57:48 +0000 (UTC) From: Alex Williamson To: robin@streamhpc.com, mst@redhat.com, marcel.apfelbaum@gmail.com Cc: Alex Williamson , qemu-devel@nongnu.org Subject: [RFC PATCH 2/4] vfio: Implement a common device info helper Date: Fri, 19 May 2023 15:57:37 -0600 Message-Id: <20230519215739.402729-3-alex.williamson@redhat.com> In-Reply-To: <20230519215739.402729-1-alex.williamson@redhat.com> References: <20230519215739.402729-1-alex.williamson@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass client-ip=170.10.129.124; envelope-from=alex.williamson@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org A common helper implementing the realloc algorithm for handling capabilities. Signed-off-by: Alex Williamson Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Cédric Le Goater --- hw/s390x/s390-pci-vfio.c | 37 ++++------------------------ hw/vfio/common.c | 46 ++++++++++++++++++++++++++--------- include/hw/vfio/vfio-common.h | 1 + 3 files changed, 41 insertions(+), 43 deletions(-) diff --git a/hw/s390x/s390-pci-vfio.c b/hw/s390x/s390-pci-vfio.c index f51190d4662f..66e8ec3bdef9 100644 --- a/hw/s390x/s390-pci-vfio.c +++ b/hw/s390x/s390-pci-vfio.c @@ -289,38 +289,11 @@ static void s390_pci_read_pfip(S390PCIBusDevice *pbdev, memcpy(pbdev->zpci_fn.pfip, cap->pfip, CLP_PFIP_NR_SEGMENTS); } -static struct vfio_device_info *get_device_info(S390PCIBusDevice *pbdev, - uint32_t argsz) +static struct vfio_device_info *get_device_info(S390PCIBusDevice *pbdev); { - struct vfio_device_info *info = g_malloc0(argsz); - VFIOPCIDevice *vfio_pci; - int fd; + VFIOPCIDevice *vfio_pci = container_of(pbdev->pdev, VFIOPCIDevice, pdev); - vfio_pci = container_of(pbdev->pdev, VFIOPCIDevice, pdev); - fd = vfio_pci->vbasedev.fd; - - /* - * If the specified argsz is not large enough to contain all capabilities - * it will be updated upon return from the ioctl. Retry until we have - * a big enough buffer to hold the entire capability chain. On error, - * just exit and rely on CLP defaults. - */ -retry: - info->argsz = argsz; - - if (ioctl(fd, VFIO_DEVICE_GET_INFO, info)) { - trace_s390_pci_clp_dev_info(vfio_pci->vbasedev.name); - g_free(info); - return NULL; - } - - if (info->argsz > argsz) { - argsz = info->argsz; - info = g_realloc(info, argsz); - goto retry; - } - - return info; + return vfio_get_device_info(vfio_pci->vbasedev.fd); } /* @@ -335,7 +308,7 @@ bool s390_pci_get_host_fh(S390PCIBusDevice *pbdev, uint32_t *fh) assert(fh); - info = get_device_info(pbdev, sizeof(*info)); + info = get_device_info(pbdev); if (!info) { return false; } @@ -356,7 +329,7 @@ void s390_pci_get_clp_info(S390PCIBusDevice *pbdev) { g_autofree struct vfio_device_info *info = NULL; - info = get_device_info(pbdev, sizeof(*info)); + info = get_device_info(pbdev); if (!info) { return; } diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 78358ede2764..ed142296e9fe 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -2843,11 +2843,35 @@ void vfio_put_group(VFIOGroup *group) } } +struct vfio_device_info *vfio_get_device_info(int fd) +{ + struct vfio_device_info *info; + uint32_t argsz = sizeof(*info); + + info = g_malloc0(argsz); + +retry: + info->argsz = argsz; + + if (ioctl(fd, VFIO_DEVICE_GET_INFO, info)) { + g_free(info); + return NULL; + } + + if (info->argsz > argsz) { + argsz = info->argsz; + info = g_realloc(info, argsz); + goto retry; + } + + return info; +} + int vfio_get_device(VFIOGroup *group, const char *name, VFIODevice *vbasedev, Error **errp) { - struct vfio_device_info dev_info = { .argsz = sizeof(dev_info) }; - int ret, fd; + g_autofree struct vfio_device_info *info = NULL; + int fd; fd = ioctl(group->fd, VFIO_GROUP_GET_DEVICE_FD, name); if (fd < 0) { @@ -2859,11 +2883,11 @@ int vfio_get_device(VFIOGroup *group, const char *name, return fd; } - ret = ioctl(fd, VFIO_DEVICE_GET_INFO, &dev_info); - if (ret) { + info = vfio_get_device_info(fd); + if (!info) { error_setg_errno(errp, errno, "error getting device info"); close(fd); - return ret; + return -1; } /* @@ -2891,14 +2915,14 @@ int vfio_get_device(VFIOGroup *group, const char *name, vbasedev->group = group; QLIST_INSERT_HEAD(&group->device_list, vbasedev, next); - vbasedev->num_irqs = dev_info.num_irqs; - vbasedev->num_regions = dev_info.num_regions; - vbasedev->flags = dev_info.flags; + vbasedev->num_irqs = info->num_irqs; + vbasedev->num_regions = info->num_regions; + vbasedev->flags = info->flags; + + trace_vfio_get_device(name, info->flags, info->num_regions, info->num_irqs); - trace_vfio_get_device(name, dev_info.flags, dev_info.num_regions, - dev_info.num_irqs); + vbasedev->reset_works = !!(info->flags & VFIO_DEVICE_FLAGS_RESET); - vbasedev->reset_works = !!(dev_info.flags & VFIO_DEVICE_FLAGS_RESET); return 0; } diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index eed244f25f34..a8dcda592c08 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -212,6 +212,7 @@ void vfio_region_finalize(VFIORegion *region); void vfio_reset_handler(void *opaque); VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp); void vfio_put_group(VFIOGroup *group); +struct vfio_device_info *vfio_get_device_info(int fd); int vfio_get_device(VFIOGroup *group, const char *name, VFIODevice *vbasedev, Error **errp); From patchwork Fri May 19 21:57:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 13248905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 58292C77B7A for ; Fri, 19 May 2023 21:59:00 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q086n-0005yv-K7; Fri, 19 May 2023 17:57:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q086l-0005x7-Lr for qemu-devel@nongnu.org; Fri, 19 May 2023 17:57:55 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q086k-0004FL-4Q for qemu-devel@nongnu.org; Fri, 19 May 2023 17:57:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684533473; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ppsSqBFmd85MY/HWcySwkiiUV0ay/ybg6reH9NMFing=; b=a0wjkSifUsoj7Cntn9saJkZrDlmzM3BmKyKjnQeO1+zE9WpdvXuC1R0/1rNVLHIU8VXkv6 Mn5gbHS1t8+QnWPfq2pFXtRWOwgT8eTER3ucPmvrZOOocV+YAZWK87q4Tm2Y5XPwtQN6ea hLpnuFrMtWr3WpRem9Fy+xG08DqdEv4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-135-QJ1nuUh6M9Cj3u7yf_XYhg-1; Fri, 19 May 2023 17:57:50 -0400 X-MC-Unique: QJ1nuUh6M9Cj3u7yf_XYhg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AE58A101A531; Fri, 19 May 2023 21:57:49 +0000 (UTC) Received: from omen.home.shazbot.org (unknown [10.22.10.168]) by smtp.corp.redhat.com (Postfix) with ESMTP id 52CA6C0004B; Fri, 19 May 2023 21:57:49 +0000 (UTC) From: Alex Williamson To: robin@streamhpc.com, mst@redhat.com, marcel.apfelbaum@gmail.com Cc: Alex Williamson , qemu-devel@nongnu.org Subject: [RFC PATCH 3/4] pcie: Add a PCIe capability version helper Date: Fri, 19 May 2023 15:57:38 -0600 Message-Id: <20230519215739.402729-4-alex.williamson@redhat.com> In-Reply-To: <20230519215739.402729-1-alex.williamson@redhat.com> References: <20230519215739.402729-1-alex.williamson@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass client-ip=170.10.129.124; envelope-from=alex.williamson@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Report the PCIe capability version for a device Signed-off-by: Alex Williamson --- hw/pci/pcie.c | 7 +++++++ include/hw/pci/pcie.h | 1 + 2 files changed, 8 insertions(+) diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c index b8c24cf45f7e..b7f107ed8dd4 100644 --- a/hw/pci/pcie.c +++ b/hw/pci/pcie.c @@ -274,6 +274,13 @@ uint8_t pcie_cap_get_type(const PCIDevice *dev) PCI_EXP_FLAGS_TYPE) >> PCI_EXP_FLAGS_TYPE_SHIFT; } +uint8_t pcie_cap_get_version(const PCIDevice *dev) +{ + uint32_t pos = dev->exp.exp_cap; + assert(pos > 0); + return pci_get_word(dev->config + pos + PCI_EXP_FLAGS) & PCI_EXP_FLAGS_VERS; +} + /* MSI/MSI-X */ /* pci express interrupt message number */ /* 7.8.2 PCI Express Capabilities Register: Interrupt Message Number */ diff --git a/include/hw/pci/pcie.h b/include/hw/pci/pcie.h index 3cc2b159570f..51ab57bc3c50 100644 --- a/include/hw/pci/pcie.h +++ b/include/hw/pci/pcie.h @@ -93,6 +93,7 @@ void pcie_cap_exit(PCIDevice *dev); int pcie_endpoint_cap_v1_init(PCIDevice *dev, uint8_t offset); void pcie_cap_v1_exit(PCIDevice *dev); uint8_t pcie_cap_get_type(const PCIDevice *dev); +uint8_t pcie_cap_get_version(const PCIDevice *dev); void pcie_cap_flags_set_vector(PCIDevice *dev, uint8_t vector); uint8_t pcie_cap_flags_get_vector(PCIDevice *dev); From patchwork Fri May 19 21:57:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 13248902 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 192A0C77B75 for ; Fri, 19 May 2023 21:58:39 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q086o-0005zD-6p; Fri, 19 May 2023 17:57:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q086m-0005yK-GP for qemu-devel@nongnu.org; Fri, 19 May 2023 17:57:56 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q086l-0004FT-1u for qemu-devel@nongnu.org; Fri, 19 May 2023 17:57:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684533474; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fgLJMJytCtabs3p65dU+5yU7aU1Lzei6dYgf+/EPmqw=; b=SKNHp9vrxIKmdZuNPM8S8tKrHRa1yakv/M9E5BM6oTZVzskPR271d13yDI9MrsRPi/St8l GGBi/Cz48gV14ch7TRIPTOPAnyaWQEChyJNydVnrRDIWbBVi4aUOHW/YCMnu5A2Q8gwfAm X9RdGoNtaCb0E2EWuIAnK01MCSueRfo= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-205-yoCbIoAGMYi19gyO_PTBUQ-1; Fri, 19 May 2023 17:57:50 -0400 X-MC-Unique: yoCbIoAGMYi19gyO_PTBUQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3042A2A5954B; Fri, 19 May 2023 21:57:50 +0000 (UTC) Received: from omen.home.shazbot.org (unknown [10.22.10.168]) by smtp.corp.redhat.com (Postfix) with ESMTP id C854AC0004B; Fri, 19 May 2023 21:57:49 +0000 (UTC) From: Alex Williamson To: robin@streamhpc.com, mst@redhat.com, marcel.apfelbaum@gmail.com Cc: Alex Williamson , qemu-devel@nongnu.org Subject: [RFC PATCH 4/4] vfio/pci: Enable AtomicOps completers on root ports Date: Fri, 19 May 2023 15:57:39 -0600 Message-Id: <20230519215739.402729-5-alex.williamson@redhat.com> In-Reply-To: <20230519215739.402729-1-alex.williamson@redhat.com> References: <20230519215739.402729-1-alex.williamson@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Received-SPF: pass client-ip=170.10.129.124; envelope-from=alex.williamson@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org For cold-plug configurations where the vfio-pci device reports enabled support for PCIe AtomicOps completers and we have a compatible VM configuration, mirror the host AtomicOps completer support in the virtual root port. Signed-off-by: Alex Williamson --- hw/vfio/pci.c | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index bf27a3990564..61d045785bf8 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -1826,6 +1826,70 @@ static void vfio_add_emulated_long(VFIOPCIDevice *vdev, int pos, vfio_set_long_bits(vdev->emulated_config_bits + pos, mask, mask); } +/* + * For a cold-plugged device connected directly to a root port where + * vfio-pci reports AtomicOp completer support on the host, twiddle + * bits in the emulated root port to reflect this capability. + */ +static void vfio_pci_enable_rp_atomics(VFIOPCIDevice *vdev) +{ + struct vfio_device_info_cap_pci_atomic_comp *cap; + g_autofree struct vfio_device_info *info = NULL; + struct vfio_info_cap_header *hdr; + PCIDevice *parent; + uint32_t mask = 0; + PCIBus *bus; + + /* + * XXX + * - Should this also be restricted to TYPE_VFIO_PCI_NOHOTPLUG devices? + * - Should we clear the capability on exit? + */ + if (vdev->pdev.qdev.hotplugged) { + return; + } + + info = vfio_get_device_info(vdev->vbasedev.fd); + if (!info) { + return; + } + + hdr = vfio_get_device_info_cap(info, VFIO_DEVICE_INFO_CAP_PCI_ATOMIC_COMP); + if (!hdr) { + return; + } + + cap = (void *)hdr; + if (cap->flags & VFIO_PCI_ATOMIC_COMP32) { + mask |= PCI_EXP_DEVCAP2_ATOMIC_COMP32; + } + if (cap->flags & VFIO_PCI_ATOMIC_COMP64) { + mask |= PCI_EXP_DEVCAP2_ATOMIC_COMP64; + } + if (cap->flags & VFIO_PCI_ATOMIC_COMP128) { + mask |= PCI_EXP_DEVCAP2_ATOMIC_COMP128; + } + + if (!mask) { + return; + } + + bus = pci_get_bus(&vdev->pdev); + parent = bus->parent_dev; + + if (!parent || !pci_is_express(parent) || !parent->exp.exp_cap) { + return; + } + + if (pcie_cap_get_type(parent) != PCI_EXP_TYPE_ROOT_PORT || + pcie_cap_get_version(parent) != PCI_EXP_FLAGS_VER2) { + return; + } + + pci_long_test_and_set_mask(parent->config + parent->exp.exp_cap + + PCI_EXP_DEVCAP2, mask); +} + static int vfio_setup_pcie_cap(VFIOPCIDevice *vdev, int pos, uint8_t size, Error **errp) { @@ -1929,6 +1993,8 @@ static int vfio_setup_pcie_cap(VFIOPCIDevice *vdev, int pos, uint8_t size, QEMU_PCI_EXP_LNKCAP_MLS(QEMU_PCI_EXP_LNK_2_5GT), ~0); vfio_add_emulated_word(vdev, pos + PCI_EXP_LNKCTL, 0, ~0); } + + vfio_pci_enable_rp_atomics(vdev); } /*